From: Jan Wieck <JanWieck@Yahoo.com>
To: olly@lfix.co.uk
Cc: Josh Berkus <josh@agliodbs.com>
Cc: Adam Lancaster <alancaster@preventsys.com>
Cc: pgsql-sql@postgresql.org
Subject: Re: Possible rounding error of large float values?
Date: Tue, 24 Aug 2004 20:39:22 -0400
Message-ID: <412BDFBA.7080405@Yahoo.com> (raw)
In-Reply-To: <1093378875.15248.39.camel@linda>
References: <8758F8D58219684FAB0239EE8967048A010807@calculon.preventsys.com>
<200408241252.30182.josh@agliodbs.com>
<1093378875.15248.39.camel@linda>
On 8/24/2004 4:21 PM, Oliver Elphick wrote:
> On Tue, 2004-08-24 at 20:52, Josh Berkus wrote:
>> Adam,
>>
>> > 9223372036854775807
>> >
>> > It gets selected out as:
>> >
>> > 9.22337203685478E18
>>
>> This is a property of FLOAT data types. They round.
>>
>> > Which appears to be rounded. When we cast it to numeric type we get:
>> >
>> > 9223372036854780000
>> >
>> > Which also is rounded. It is still possible to find the row using the
>> > original value
>>
>> Hmmm ... is 15 digits the limit of NUMERIC? It may be.
The original numeric code I presented contained even log(), ln() and
trigonometric functions that produced results of up to 1,000 digits
precision. They wheren't considered usefull, so they didn't survive, but
numeric is that good after all.
Jan
>
> It must be the limit of float. Numeric can't produce any more than it
> was given and it was cast from float.
>
> bray=# select 653596708775675750507850507570708696432 ::numeric;
> numeric
> -----------------------------------------
> 653596708775675750507850507570708696432
> (1 row)
>
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-sql@postgresql.org
Cc: JanWieck@Yahoo.com, olly@lfix.co.uk, josh@agliodbs.com, alancaster@preventsys.com
Subject: Re: Possible rounding error of large float values?
In-Reply-To: <412BDFBA.7080405@Yahoo.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox