pg.ddx.io  pgsql-php@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Jasen Betts <jasen@xnet.co.nz>
To: pgsql-php@postgresql.org
Subject: Re: large resultset
Date: 15 Jun 2010 12:09:25 GMT
Message-ID: <hv7qhl$ogq$1@reversiblemaps.ath.cx> (raw)
References: <AANLkTinXSz6PsoTyPhryAHsN3V-M0v-m3sTCLTJSJttO@mail.gmail.com>
	<AANLkTimP7qSASY3dRA09p6e35CzRLo8pOyZ6J5_5Bisx@mail.gmail.com>
	<8f468dc246fb6075f2a1f46b09f71a6d@xs4all.nl>

On 2010-06-15, vinny <vinny@xs4all.nl> wrote:
>
> On Tue, 15 Jun 2010 10:33:05 +0100, Thom Brown <thombrown@gmail.com>
> wrote:
>> On 15 June 2010 09:31, AI Rumman  wrote:
>>  How to return large resutlset (almost 2 millions record) in php?
>> 
>> Presumably this is so people can download results rather than display
>> them in a browser?
>> 
>> Here's a basic a raw version of what you can do:
>> 
>> $db = pg_connect($connection_string);
>> 
>> $results = pg_query($db, $query);
>> 
>> while ($result = pg_fetch_array($results)
>> {
>>     echo implode("|", $result) . "n";
>> }
>> 
>> pg_close($db);
>> 
>> Obviously that wouldn't do by itself, but it's quite simple to loop over
>> a result set.
>> 
>> Regards
>> 
>> Thom
>> 
>
> If that's the case surely you'd use some SQL to merge the data into one
> long string and return it in a single record.
> Looping over two million results is going to take a while,

yeah, so is transmitting the bytes.

> not to mention quite a bit of memory.

yeah, libpq is going to want to load the result-set into memory.
you'll probably hit PHPs memory limit and it'll be game over.

The only pure php solution is to use a cursor. (seing as pg_get_line
is not avaulable)

OTOH you can do copy like this...

  $c=escapeshellarg($connstr);
  $q=escapeshellarg($query);
  system ("psql $c -c 'copy ( '$q' ) to stdout with csv header'");
  
needs *nix based server and pgversion >= 8.3 (I think).

You don't get to choose the format beyond what pg supports, but I
think you can tweak encoding on the command-line if needed.
OTOH you could use ICONV for that. 



view thread (14+ messages)  latest in thread

Message-ID: <hv7qhl$ogq$1@reversiblemaps.ath.cx>
Permalink:  ../hv7qhl$ogq$1@reversiblemaps.ath.cx/
Also on:    postgresql.org/message-id/hv7qhl$ogq$1@reversiblemaps.ath.cx

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-php@postgresql.org
  Cc: jasen@xnet.co.nz
  Subject: Re: large resultset
  In-Reply-To: <hv7qhl$ogq$1@reversiblemaps.ath.cx>
  References: <AANLkTinXSz6PsoTyPhryAHsN3V-M0v-m3sTCLTJSJttO@mail.gmail.com> <AANLkTimP7qSASY3dRA09p6e35CzRLo8pOyZ6J5_5Bisx@mail.gmail.com> <8f468dc246fb6075f2a1f46b09f71a6d@xs4all.nl> <hv7qhl$ogq$1@reversiblemaps.ath.cx>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox