public inbox for [email protected]  

help / color / mirror / Atom feed

From: Joel Jacobson <[email protected]>
To: Dean Rasheed <[email protected]>
Cc: Dagfinn Ilmari Mannsåker <[email protected]>
Cc: pgsql-hackers <[email protected]>
Subject: Re: Optimize numeric multiplication for one and two base-NBASE digit multiplicands.
Date: Tue, 02 Jul 2024 21:55:19 +0200
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<CAEZATCV2qPTGo2Fd8xDs06Q7iU5aorgSa9+Fw9zkuQv1y15rcw@mail.gmail.com>
	<[email protected]>
	<CAEZATCWEou=NjpvS-r1WxymbAYihsdKQoQBj1HfBeocoNcv5Ww@mail.gmail.com>
	<[email protected]>
	<[email protected]>
	<CAEZATCW+N9qaE=5_1xcB0isMrirojN=Bd0yY4Pw67iji4swYiQ@mail.gmail.com>
	<[email protected]>
	<[email protected]>

On Tue, Jul 2, 2024, at 20:53, Joel Jacobson wrote:
> Trying to wrap my head around what could cause this.
>
> It's rounding down instead of up, and these cases all end with decimal .500XXXX.

Interesting, I actually think there is a bug in the normal mul_var() code.
Found a case that rounds down, when it should round up:

Calling mul_var() with:
var1=51.2945442386599
var2=0.828548712212
rscale=0

returns 42, but I think it should return 43,
since 51.2945442386599*0.828548712212=42.5000285724431241296446988

But maybe this is expected and OK, having to do with MUL_GUARD_DIGITS?

/Joel

view thread (13+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected]
  Subject: Re: Optimize numeric multiplication for one and two base-NBASE digit multiplicands.
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox