From: Alvaro Herrera <alvherre@alvh.no-ip.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: libpq: Improve idle state handling in pipeline mode
Date: Tue, 05 Jul 2022 12:38:43 +0000
Message-ID: <E1o8hpC-001JON-5U@gemulon.postgresql.org> (raw)
libpq: Improve idle state handling in pipeline mode
We were going into IDLE state too soon when executing queries via
PQsendQuery in pipeline mode, causing several scenarios to misbehave in
different ways -- most notably, as reported by Daniele Varrazzo, that a
warning message is produced by libpq:
message type 0x33 arrived from server while idle
But it is also possible, if queries are sent and results consumed not in
lockstep, for the expected mediating NULL result values from PQgetResult
to be lost (a problem which has not been reported, but which is more
serious).
Fix this by introducing two new concepts: one is a command queue element
PGQUERY_CLOSE to tell libpq to wait for the CloseComplete server
response to the Close message that is sent by PQsendQuery. Because the
application is not expecting any PGresult from this, the mechanism to
consume it is a bit hackish.
The other concept, authored by Horiguchi-san, is a PGASYNC_PIPELINE_IDLE
state for libpq's state machine to differentiate "really idle" from
merely "the idle state that occurs in between reading results from the
server for elements in the pipeline". This makes libpq not go fully
IDLE when the libpq command queue contains entries; in normal cases, we
only go IDLE once at the end of the pipeline, when the server response
to the final SYNC message is received. (However, there are corner cases
it doesn't fix, such as terminating the query sequence by
PQsendFlushRequest instead of PQpipelineSync; this sort of scenario is
what requires PGQUERY_CLOSE bit above.)
This last bit helps make the libpq state machine clearer; in particular
we can get rid of an ugly hack in pqParseInput3 to avoid considering
IDLE as such when the command queue contains entries.
A new test mode is added to libpq_pipeline.c to tickle some related
problematic cases.
Reported-by: Daniele Varrazzo <daniele.varrazzo@gmail.com>
Co-authored-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Discussion: https://postgr.es/m/CA+mi_8bvD0_CW3sumgwPvWdNzXY32itoG_16tDYRu_1S2gV2iw@mail.gmail.com
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/054325c5eeb3140a067ba66735c3d811163ecd6a
Modified Files
--------------
src/interfaces/libpq/fe-exec.c | 112 +++++++++--
src/interfaces/libpq/fe-protocol3.c | 30 +--
src/interfaces/libpq/libpq-int.h | 6 +-
src/test/modules/libpq_pipeline/libpq_pipeline.c | 215 ++++++++++++++++++++-
.../modules/libpq_pipeline/t/001_libpq_pipeline.pl | 3 +-
.../libpq_pipeline/traces/pipeline_idle.trace | 93 +++++++++
6 files changed, 424 insertions(+), 35 deletions(-)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-committers@postgresql.org
Cc: alvherre@alvh.no-ip.org
Subject: Re: pgsql: libpq: Improve idle state handling in pipeline mode
In-Reply-To: <E1o8hpC-001JON-5U@gemulon.postgresql.org>
References: <E1o8hpC-001JON-5U@gemulon.postgresql.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox