Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 33661632CC3 for ; Tue, 15 Jun 2010 09:09:47 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 74117-03 for ; Tue, 15 Jun 2010 12:09:38 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from smtpcorp.com (smtpcorp.com [67.192.120.76]) by mail.postgresql.org (Postfix) with ESMTP id F0D4D632C95 for ; Tue, 15 Jun 2010 09:09:39 -0300 (ADT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=smtpcorp.com; s=main; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:Message-ID:Date:Subject:From:To; bh=hFMIl/t5zDUfEIrxx7bD36BE43eL4fknTi2AZJtWwSY=; b=CmH4m6Qd7LTta0MA0wwHOoMc3gxTw4f9bGsrsIPc9kgDgFXAzGRV4qsGtry663FOU6eeDyuigRMbg0CcbSnX+vZYwQzCxKiHGyG20o8fM56tLKaea/4OieGRDByIFpuT; Received: from ip-118-90-143-50.xdsl.xnet.co.nz ([118.90.143.50] helo=gonzo) by smtpcorp.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.70) id 1OOUxV-0006g5-3k; mid 144 for pgsql-php@postgresql.org; Tue, 15 Jun 2010 12:09:13 +0000 Received: from news by gonzo with local (Exim 4.69) (envelope-from ) id 1OOUxJ-0006X6-FY for pgsql-php@postgresql.org; Wed, 16 Jun 2010 00:09:25 +1200 To: pgsql-php@postgresql.org Path: not-for-mail From: Jasen Betts Newsgroups: gmane.comp.db.postgresql.php Subject: Re: large resultset Date: 15 Jun 2010 12:09:25 GMT Organization: Dis (not Dat) Organisation Lines: 56 Message-ID: References: <8f468dc246fb6075f2a1f46b09f71a6d@xs4all.nl> NNTP-Posting-Host: localhost Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Trace: reversiblemaps.ath.cx 1276603765 25114 127.0.0.1 (15 Jun 2010 12:09:25 GMT) X-Face: ?)Aw4rXwN5u0~$nqKj`xPz>xHCwgi^q+^?Ri*+R(&uv2=E1Q0Zk(>h!~o2ID@6{uf8s; a +M[5[U[QT7xFN%^gR"=tuJw%TXXR'Fp~W; (T"1(739R%m0Yyyv*gkGoPA.$b,D.w:z+<'"=-lVT?6 {T?=R^:W5g|E2#EhjKCa+nt":4b}dU7GYB*HBxn&Td$@f%.kl^:7X8rQWd[NTc"P"u6nkisze/Q; 8 "9Z{peQF,w)7UjV$c|RO/mQW/NMgWfr5*$-Z%u46"/00mx-,\R'fLPe.)^ User-Agent: slrn/pre1.0.0-2 (Linux) X-Smtpcorp-Track: 4508952.1.56577883 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9 X-Spam-Level: X-Archive-Number: 201006/8 X-Sequence-Number: 4147 On 2010-06-15, vinny wrote: > > On Tue, 15 Jun 2010 10:33:05 +0100, Thom Brown > wrote: >> On 15 June 2010 09:31, AI Rumman wrote: >> How to return large resutlset (almost 2 millions record) in php? >> >> Presumably this is so people can download results rather than display >> them in a browser? >> >> Here's a basic a raw version of what you can do: >> >> $db = pg_connect($connection_string); >> >> $results = pg_query($db, $query); >> >> while ($result = pg_fetch_array($results) >> { >>     echo implode("|", $result) . "n"; >> } >> >> pg_close($db); >> >> Obviously that wouldn't do by itself, but it's quite simple to loop over >> a result set. >> >> Regards >> >> Thom >> > > If that's the case surely you'd use some SQL to merge the data into one > long string and return it in a single record. > Looping over two million results is going to take a while, yeah, so is transmitting the bytes. > not to mention quite a bit of memory. yeah, libpq is going to want to load the result-set into memory. you'll probably hit PHPs memory limit and it'll be game over. The only pure php solution is to use a cursor. (seing as pg_get_line is not avaulable) OTOH you can do copy like this... $c=escapeshellarg($connstr); $q=escapeshellarg($query); system ("psql $c -c 'copy ( '$q' ) to stdout with csv header'"); needs *nix based server and pgversion >= 8.3 (I think). You don't get to choose the format beyond what pg supports, but I think you can tweak encoding on the command-line if needed. OTOH you could use ICONV for that.