To: pgsql-interfaces@postgresql.org
From: Bob Gilson <bgilson@jabber.com>
Subject: Re: cursor interface to libpq limited by virtual memory
Date: Thu, 29 Apr 2004 12:12:04 -0600
Lines: 28
Message-ID: <c6rgms$1tv$1@sea.gmane.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040311

Below is a post from 2000. It describes a limitation processing large 
selects with libpq. Does this limitation still exist?

Thanks in advance,
Bob Gilson


Kirby Bohling wrote:
 >>I am trying to run a select statement, and I keep running out of
 >>memory. After a little investigate, I found
 >>that libpq appears to get the entire result set at once.


The bottleneck here is mainly that libpq's API is defined in terms of
providing random access to a result set, no matter how large --- so
libpq has to buffer the whole result set in client memory.

Aside from random access there are also error-reporting issues.
Currently libpq guarantees to tell you about any errors encountered
during a query before you start to read result rows.  That guarantee
wouldn't hold in a streaming-results scenario.

These issues have been discussed quite a few times before --- see the
pg-interfaces archives.  I think everyone agrees that it'd be a good
idea to have a streamable libpq interface, but no one's stepped up to
the plate to define or implement one...

			regards, tom lane