From: Josh Berkus <josh@agliodbs.com>
To: Adam Lancaster <alancaster@preventsys.com>
To: pgsql-sql@postgresql.org
Subject: Re: Possible rounding error of large float values?
Date: Tue, 24 Aug 2004 12:52:30 -0700
Message-ID: <200408241252.30182.josh@agliodbs.com> (raw)
In-Reply-To: <8758F8D58219684FAB0239EE8967048A010807@calculon.preventsys.com>
References: <8758F8D58219684FAB0239EE8967048A010807@calculon.preventsys.com>
Adam,
> 9223372036854775807
>
> It gets selected out as:
>
> 9.22337203685478E18
This is a property of FLOAT data types. They round.
> Which appears to be rounded. When we cast it to numeric type we get:
>
> 9223372036854780000
>
> Which also is rounded. It is still possible to find the row using the
> original value
Hmmm ... is 15 digits the limit of NUMERIC? It may be.
> Is this expected behavior?
Yes.
--
Josh Berkus
Aglio Database Solutions
San Francisco
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-sql@postgresql.org
Cc: josh@agliodbs.com, alancaster@preventsys.com
Subject: Re: Possible rounding error of large float values?
In-Reply-To: <200408241252.30182.josh@agliodbs.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox