From: Bob Gilson <bgilson@jabber.com>
To: pgsql-interfaces@postgresql.org
Subject: Re: cursor interface to libpq limited by virtual memory
Date: Thu, 29 Apr 2004 12:12:04 -0600
Message-ID: <c6rgms$1tv$1@sea.gmane.org> (raw)
Below is a post from 2000. It describes a limitation processing large
selects with libpq. Does this limitation still exist?
Thanks in advance,
Bob Gilson
Kirby Bohling wrote:
>>I am trying to run a select statement, and I keep running out of
>>memory. After a little investigate, I found
>>that libpq appears to get the entire result set at once.
The bottleneck here is mainly that libpq's API is defined in terms of
providing random access to a result set, no matter how large --- so
libpq has to buffer the whole result set in client memory.
Aside from random access there are also error-reporting issues.
Currently libpq guarantees to tell you about any errors encountered
during a query before you start to read result rows. That guarantee
wouldn't hold in a streaming-results scenario.
These issues have been discussed quite a few times before --- see the
pg-interfaces archives. I think everyone agrees that it'd be a good
idea to have a streamable libpq interface, but no one's stepped up to
the plate to define or implement one...
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-interfaces@postgresql.org
Cc: bgilson@jabber.com
Subject: Re: cursor interface to libpq limited by virtual memory
In-Reply-To: <c6rgms$1tv$1@sea.gmane.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox