From: Oliver Elphick <olly@lfix.co.uk>
To: Josh Berkus <josh@agliodbs.com>
Cc: Adam Lancaster <alancaster@preventsys.com>
Cc: pgsql-sql@postgresql.org
Subject: Re: Possible rounding error of large float values?
Date: Tue, 24 Aug 2004 21:21:15 +0100
Message-ID: <1093378875.15248.39.camel@linda> (raw)
In-Reply-To: <200408241252.30182.josh@agliodbs.com>
References: <8758F8D58219684FAB0239EE8967048A010807@calculon.preventsys.com>
<200408241252.30182.josh@agliodbs.com>
On Tue, 2004-08-24 at 20:52, Josh Berkus wrote:
> Adam,
>
> > 9223372036854775807
> >
> > It gets selected out as:
> >
> > 9.22337203685478E18
>
> This is a property of FLOAT data types. They round.
>
> > Which appears to be rounded. When we cast it to numeric type we get:
> >
> > 9223372036854780000
> >
> > Which also is rounded. It is still possible to find the row using the
> > original value
>
> Hmmm ... is 15 digits the limit of NUMERIC? It may be.
It must be the limit of float. Numeric can't produce any more than it
was given and it was cast from float.
bray=# select 653596708775675750507850507570708696432 ::numeric;
numeric
-----------------------------------------
653596708775675750507850507570708696432
(1 row)
--
Oliver Elphick olly@lfix.co.uk
Isle of Wight http://www.lfix.co.uk/oliver
GPG: 1024D/A54310EA 92C8 39E7 280E 3631 3F0E 1EC0 5664 7A2F A543 10EA
========================================
"I saw in the night visions, and, behold, one like the
Son of man came with the clouds of heaven, and came to
the Ancient of days, and they brought him near before
him. And there was given him dominion, and glory, and
a kingdom, that all people, nations, and languages,
should serve him; his dominion is an everlasting
dominion, which shall not pass away, and his kingdom
that which shall not be destroyed."
Daniel 7:13,14
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-sql@postgresql.org
Cc: olly@lfix.co.uk, josh@agliodbs.com, alancaster@preventsys.com
Subject: Re: Possible rounding error of large float values?
In-Reply-To: <1093378875.15248.39.camel@linda>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox