From: Tom Lane <tgl@sss.pgh.pa.us>
To: Mike Beaton <mjsbeaton@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Correct use of cursors for very large result sets in Postgres
Date: Sat, 18 Feb 2017 12:43:49 -0500
Message-ID: <26739.1487439829@sss.pgh.pa.us> (raw)
In-Reply-To: <CAHzAAWQhQKsE2isFr6bGL-Q06CTQSg6Sjd_+Nt-4h9Gqu0aVHg@mail.gmail.com>
References: <CAHzAAWR4Hr8OQdjCuhDNsxTz3=qit4ufObSH=bJP9xxt2NLwLg@mail.gmail.com>
<CAHzAAWRScAvevHu6AyK0ZZxnWo+SehXO9u51OSFvJEj9=61fXg@mail.gmail.com>
<21063.1487349580@sss.pgh.pa.us>
<CAHzAAWQhQKsE2isFr6bGL-Q06CTQSg6Sjd_+Nt-4h9Gqu0aVHg@mail.gmail.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-performance>
Mike Beaton <mjsbeaton@gmail.com> writes:
> One outstanding question I have. Based on a lot of helpful responses given
> to the SO question I can now test and see what disk buffers are generated
> (by setting `log_temp_files` to `0` and then `tail -f log`), as well as how
> long it takes for results to start arriving.
> With a large (10,000,000 row) test table, if I do `SELECT * FROM table` on
> psql it starts to return results immediately with no disk buffer. If I do
> `FETCH ALL FROM cursortotable` on psql it takes about 7.5 seconds to start
> returning results, and generates a 14MB buffer. If I do `SELECT * FROM
> table` on a correctly coded streaming client, it also starts to return
> results immediately with no disk buffer. But if I do `FETCH ALL FROM
> cursortotable` from my streaming client, it takes about 1.5 seconds for
> results to start coming... but again with no disk buffer, as hoped
Seems odd. Is your cursor just on "SELECT * FROM table", or is there
some processing in there you're not mentioning? Maybe it's a cursor
WITH HOLD and you're exiting the source transaction?
regards, tom lane
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: tgl@sss.pgh.pa.us, mjsbeaton@gmail.com
Subject: Re: Correct use of cursors for very large result sets in Postgres
In-Reply-To: <26739.1487439829@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox