public inbox for [email protected]  

help / color / mirror / Atom feed

Numeric Scale Differences
3+ messages / 2 participants
[nested] [flat]

* Numeric Scale Differences
@ 2021-10-21 13:47  Nuno Sousa <[email protected]>
  0 siblings, 1 reply; 3+ messages in thread

From: Nuno Sousa @ 2021-10-21 13:47 UTC (permalink / raw)
  To: [email protected]

Hi all,

I've run into an interesting issue when trying to do numeric math. I
expected the following to return the same number of decimal digits in the
fractional part:

select scale(1/0.6::numeric), scale(1/6::numeric);

But it returns a scale of 16 and 20, respectively.

Any idea why?
--
Nuno Sousa


^ permalink  raw  reply  [nested|flat] 3+ messages in thread

* Re: Numeric Scale Differences
@ 2021-10-21 14:06  Tom Lane <[email protected]>
  parent: Nuno Sousa <[email protected]>
  0 siblings, 1 reply; 3+ messages in thread

From: Tom Lane @ 2021-10-21 14:06 UTC (permalink / raw)
  To: Nuno Sousa <[email protected]>; +Cc: [email protected]

Nuno Sousa <[email protected]> writes:
> I've run into an interesting issue when trying to do numeric math. I
> expected the following to return the same number of decimal digits in the
> fractional part:

> select scale(1/0.6::numeric), scale(1/6::numeric);

Um ... why did you expect that?  The inputs to the divisions have
different scales:

# select scale(0.6::numeric), scale(6::numeric);
 scale | scale 
-------+-------
     1 |     0
(1 row)

so I find it unsurprising that the outputs do too.  Now the fact
that the output scales differ by 4 not 1 is indeed an implementation
artifact.  It stems from the numeric type working with base-10000
digits, so that scales that are multiples of 4 are most efficient,
hence division will always choose such an output scale.

			regards, tom lane





^ permalink  raw  reply  [nested|flat] 3+ messages in thread

* Re: Numeric Scale Differences
@ 2021-10-21 15:26  Nuno Sousa <[email protected]>
  parent: Tom Lane <[email protected]>
  0 siblings, 0 replies; 3+ messages in thread

From: Nuno Sousa @ 2021-10-21 15:26 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: [email protected]

Hi Tom,

Thanks for the quick response. In that case, do you think it's worth
documenting the implementation artifact behavior on the docs?
--
Nuno Sousa


On Thu, Oct 21, 2021 at 3:06 PM Tom Lane <[email protected]> wrote:

> Nuno Sousa <[email protected]> writes:
> > I've run into an interesting issue when trying to do numeric math. I
> > expected the following to return the same number of decimal digits in the
> > fractional part:
>
> > select scale(1/0.6::numeric), scale(1/6::numeric);
>
> Um ... why did you expect that?  The inputs to the divisions have
> different scales:
>
> # select scale(0.6::numeric), scale(6::numeric);
>  scale | scale
> -------+-------
>      1 |     0
> (1 row)
>
> so I find it unsurprising that the outputs do too.  Now the fact
> that the output scales differ by 4 not 1 is indeed an implementation
> artifact.  It stems from the numeric type working with base-10000
> digits, so that scales that are multiples of 4 are most efficient,
> hence division will always choose such an output scale.
>
>                         regards, tom lane
>


^ permalink  raw  reply  [nested|flat] 3+ messages in thread

end of thread, other threads:[~2021-10-21 15:26 UTC | newest]

Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-10-21 13:47 Numeric Scale Differences Nuno Sousa <[email protected]>
2021-10-21 14:06 ` Tom Lane <[email protected]>
2021-10-21 15:26   ` Nuno Sousa <[email protected]>

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox