From: Iain <iain@mst.co.jp>
To: Adam Lancaster <alancaster@preventsys.com>
To: pgsql-sql@postgresql.org
Subject: Re: Possible rounding error of large float values?
Date: Wed, 25 Aug 2004 10:32:29 +0900
Message-ID: <005b01c48a43$630db0a0$7201a8c0@mst1x5r347kymb> (raw)
References: <8758F8D58219684FAB0239EE8967048A010807@calculon.preventsys.com>
<200408241252.30182.josh@agliodbs.com>
<1093378875.15248.39.camel@linda>
# select 9223372036854775807 = 9223372036854775807::float;
?column?
----------
t
(1 row)
# select 9223372036854775807 = 9223372036854775807.0::float;
?column?
----------
t
(1 row)
# select 9223372036854775807 = 9223372036854775807.0000::numeric;
?column?
----------
t
(1 row)
This and the fact that it's still possible to find the row using the
original value would seem to indicate that the rounding is just a display
artifact..
----- Original Message -----
From: "Oliver Elphick" <olly@lfix.co.uk>
To: "Josh Berkus" <josh@agliodbs.com>
Cc: "Adam Lancaster" <alancaster@preventsys.com>; <pgsql-sql@postgresql.org>
Sent: Wednesday, August 25, 2004 5:21 AM
Subject: Re: [SQL] Possible rounding error of large float values?
> On Tue, 2004-08-24 at 20:52, Josh Berkus wrote:
> > Adam,
> >
> > > 9223372036854775807
> > >
> > > It gets selected out as:
> > >
> > > 9.22337203685478E18
> >
> > This is a property of FLOAT data types. They round.
> >
> > > Which appears to be rounded. When we cast it to numeric type we get:
> > >
> > > 9223372036854780000
> > >
> > > Which also is rounded. It is still possible to find the row using the
> > > original value
> >
> > Hmmm ... is 15 digits the limit of NUMERIC? It may be.
>
> It must be the limit of float. Numeric can't produce any more than it
> was given and it was cast from float.
>
> bray=# select 653596708775675750507850507570708696432 ::numeric;
> numeric
> -----------------------------------------
> 653596708775675750507850507570708696432
> (1 row)
>
> --
> Oliver Elphick olly@lfix.co.uk
> Isle of Wight http://www.lfix.co.uk/oliver
> GPG: 1024D/A54310EA 92C8 39E7 280E 3631 3F0E 1EC0 5664 7A2F A543 10EA
> ========================================
> "I saw in the night visions, and, behold, one like the
> Son of man came with the clouds of heaven, and came to
> the Ancient of days, and they brought him near before
> him. And there was given him dominion, and glory, and
> a kingdom, that all people, nations, and languages,
> should serve him; his dominion is an everlasting
> dominion, which shall not pass away, and his kingdom
> that which shall not be destroyed."
> Daniel 7:13,14
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-sql@postgresql.org
Cc: iain@mst.co.jp, alancaster@preventsys.com
Subject: Re: Possible rounding error of large float values?
In-Reply-To: <005b01c48a43$630db0a0$7201a8c0@mst1x5r347kymb>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox