Feedback-ID: ic6394509:Fastmail
User-Agent: Cyrus-JMAP/3.11.0-alpha0-566-g3812ddbbc-fm-20240627.001-g3812ddbb
MIME-Version: 1.0
Message-Id: <d952952f-2ba6-4e08-adbe-9e17cb26d8a4@app.fastmail.com>
In-Reply-To: 
 <CAEZATCWhg+6pFhmmeCPw8yw4c7JqwYVs-54S4kt4Jpo7ZOmc_g@mail.gmail.com>
References: 
 <CAEZATCXB+rDTuMjhK5ZxcouufigSc-X4tGJCBTMpZ3n=xxQuhg@mail.gmail.com>
 <c1ab12df-1357-4469-9761-9355015c039b@app.fastmail.com>
 <CAEZATCWhg+6pFhmmeCPw8yw4c7JqwYVs-54S4kt4Jpo7ZOmc_g@mail.gmail.com>
Date: Mon, 08 Jul 2024 12:08:10 +0200
From: "Joel Jacobson" <joel@compiler.org>
To: "Dean Rasheed" <dean.a.rasheed@gmail.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: Incorrect results from numeric round() and trunc()
Content-Type: text/plain
Archived-At: <https://www.postgresql.org/message-id/d952952f-2ba6-4e08-adbe-9e17cb26d8a4%40app.fastmail.com>
Precedence: bulk

On Mon, Jul 8, 2024, at 11:45, Dean Rasheed wrote:
> On Mon, 8 Jul 2024 at 00:40, Joel Jacobson <joel@compiler.org> wrote:
>>
>> On Sun, Jul 7, 2024, at 13:28, Dean Rasheed wrote:
>> > I've also tidied up a bit by replacing all instances of SHRT_MAX with
>> > a new constant NUMERIC_WEIGHT_MAX, whose name more accurately
>> > describes the limit, as used in various other overflow checks.
>>
>> Having thought a bit more on this, I think we probably need a
>> DEC_DIGITS sensitive definition of NUMERIC_WEIGHT_MAX,
>> since per spec the max range for numeric is 0x20000 (131072)
>> decimal digits.
>>
>
> No, the maximum weight is determined by the use of int16 to store the
> weight. Therefore if you did reduce DEC_DIGITS to 1 or 2, the number
> of decimal digits allowed before the decimal point would be reduced
> too.

OK, that can actually be seen as a feature, especially since it's
of course more likely DEC_DIGITS could increase in the future
than decrease.

For example, let's say we would double it to 8,
then if NUMERIC_WEIGHT_MAX would still be 0x7FFF (32767),
then the maximum range for numeric would increase from 131072 to 262144
decimal digits allowed before the decimal point.

LGTM.

Regards,
Joel