public inbox for [email protected]  

help / color / mirror / Atom feed

From: Nuno Sousa <[email protected]>
To: Tom Lane <[email protected]>
Cc: [email protected]
Subject: Re: Numeric Scale Differences
Date: Thu, 21 Oct 2021 16:26:26 +0100
Message-ID: <CAMZ4ZVo-gMBDOvLKBory1yeHjsgV1NjWMtc5R4sswi6vbw8SuQ@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <CAMZ4ZVqq9kHN7DJs5T=zL8jDJuk-3v2-spyYtjd8e2xj_OPM8Q@mail.gmail.com>
	<[email protected]>

Hi Tom,

Thanks for the quick response. In that case, do you think it's worth
documenting the implementation artifact behavior on the docs?
--
Nuno Sousa


On Thu, Oct 21, 2021 at 3:06 PM Tom Lane <[email protected]> wrote:

> Nuno Sousa <[email protected]> writes:
> > I've run into an interesting issue when trying to do numeric math. I
> > expected the following to return the same number of decimal digits in the
> > fractional part:
>
> > select scale(1/0.6::numeric), scale(1/6::numeric);
>
> Um ... why did you expect that?  The inputs to the divisions have
> different scales:
>
> # select scale(0.6::numeric), scale(6::numeric);
>  scale | scale
> -------+-------
>      1 |     0
> (1 row)
>
> so I find it unsurprising that the outputs do too.  Now the fact
> that the output scales differ by 4 not 1 is indeed an implementation
> artifact.  It stems from the numeric type working with base-10000
> digits, so that scales that are multiples of 4 are most efficient,
> hence division will always choose such an output scale.
>
>                         regards, tom lane
>

view thread (3+ messages)

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected]
  Subject: Re: Numeric Scale Differences
  In-Reply-To: <CAMZ4ZVo-gMBDOvLKBory1yeHjsgV1NjWMtc5R4sswi6vbw8SuQ@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox