Date: Fri, 17 Dec 2010 13:51:27 +0000
From: Nick Matheson <Nick.D.Matheson@noaa.gov>
Subject: Re: Help with bulk read performance
In-reply-to: <op.vntbn2ujeorkce@apollo13>
To: Pierre C <lists@peufeu.com>
Cc: Andy Colson <andy@squeakycode.net>,
	Dan Schaffer <Daniel.S.Schaffer@noaa.gov>, Jim Nasby <jim@nasby.net>,
	pgsql-performance@postgresql.org
Message-id: <4D0B6ADF.7050303@noaa.gov>
MIME-version: 1.0
Content-type: text/plain; charset=UTF-8; format=flowed
Content-transfer-encoding: 7BIT
User-Agent: Thunderbird 2.0.0.24 (X11/20101103)
References: <4CCECB69.7080906@noaa.gov> <4D078CD7.7030308@squeakycode.net>
	<B57C829F-DFE3-4DC7-877B-14D5B05AB4BB@nasby.net>
	<4D07928B.2070907@squeakycode.net> <4D0921D2.2030806@noaa.gov>
	<op.vntbn2ujeorkce@apollo13>

Pierre-

I agree with your observation of float <-> text conversion costs, but in 
this case Dan is talking about storing the raw float data (ie: 4 bytes 
per float) in a bytea array so there is only the conversion from java 
float[n] to java byte[4*n] which is not nearly as costly as float <-> 
text conversion (especially if you leave it in architecture byte order).

Nick
>
>> If the data are stored as a byte array but retrieve into a ResultSet, 
>> the unpacking time goes up by an order of magnitude and the
>> observed total throughput is 25 MB/s.  If the data are stored in a 
>> Postgres float array and unpacked into a byte stream, the
>> observed throughput is 20 MB/s.
>
>
> float <-> text conversions are very slow, this is in fact due to the 
> mismatch between base-2 (IEEE754) and base-10 (text) floating point 
> representation, which needs very very complex calculations.