Feedback-ID: ic6394509:Fastmail
User-Agent: Cyrus-JMAP/3.11.0-alpha0-566-g3812ddbbc-fm-20240627.001-g3812ddbb
MIME-Version: 1.0
Message-Id: <c1ab12df-1357-4469-9761-9355015c039b@app.fastmail.com>
In-Reply-To: 
 <CAEZATCXB+rDTuMjhK5ZxcouufigSc-X4tGJCBTMpZ3n=xxQuhg@mail.gmail.com>
References: 
 <CAEZATCXB+rDTuMjhK5ZxcouufigSc-X4tGJCBTMpZ3n=xxQuhg@mail.gmail.com>
Date: Mon, 08 Jul 2024 01:40:31 +0200
From: "Joel Jacobson" <joel@compiler.org>
To: "Dean Rasheed" <dean.a.rasheed@gmail.com>,
 pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: Incorrect results from numeric round() and trunc()
Content-Type: text/plain
Archived-At: <https://www.postgresql.org/message-id/c1ab12df-1357-4469-9761-9355015c039b%40app.fastmail.com>
Precedence: bulk

On Sun, Jul 7, 2024, at 13:28, Dean Rasheed wrote:
> I've also tidied up a bit by replacing all instances of SHRT_MAX with
> a new constant NUMERIC_WEIGHT_MAX, whose name more accurately
> describes the limit, as used in various other overflow checks.

Having thought a bit more on this, I think we probably need a
DEC_DIGITS sensitive definition of NUMERIC_WEIGHT_MAX,
since per spec the max range for numeric is 0x20000 (131072)
decimal digits.

Therefore, I think perhaps what we want is:

+#define NUMERIC_DSCALE_MIN                     0
+#define NUMERIC_WEIGHT_MAX                     ((0x20000/DEC_DIGITS)-1)
+#define NUMERIC_WEIGHT_MIN                     (-(NUMERIC_DSCALE_MAX+1)/DEC_DIGITS)

Maybe also 0x20000 (131072) should be a defined constant.

Regards,
Joel