pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedPATCH: Batch/pipelining support for libpq
182+ messages / 32 participants
[nested] [flat]
Commits from this thread:
2021-03-15 9aa491abbf Add libpq pipeline mode support to pgbench
2021-03-15 acb7e4eb6b Implement pipeline mode in libpq
Related threads (commitfest entry Batch (pipelined) mode for libpq/634):
libpq pipelining
* PATCH: Batch/pipelining support for libpq
@ 2016-05-23 09:19 Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-24 13:55 ` Re: PATCH: Batch/pipelining support for libpq Jim Nasby <Jim.Nasby@BlueTreble.com>
2016-10-12 11:51 ` Re: PATCH: Batch/pipelining support for libpq Shay Rojansky <roji@roji.org>
0 siblings, 3 replies; 182+ messages in thread
From: Craig Ringer @ 2016-05-23 09:19 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Michael Paquier <michael.paquier@gmail.com>; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
Hi all
Following on from the foreign table batch inserts thread[1], here's a patch
to add support for pipelining queries into asynchronous batches in libpq.
Attached, and also available at
https://github.com/2ndQuadrant/postgres/tree/dev/libpq-async-batch (subject
to rebasing and force pushes).
It's cleaned up over the draft I posted on that thread and has error
recovery implemented. I've written and included the SGML docs for it. The
test program is now pretty comprehensive, more so than for anything else in
libpq anyway. I'll submit it to the next CF as a 9.7/10.0 candidate.
I'm measuring 300x (not %) performance improvements doing batches on
servers over the Internet, so this seems pretty worthwhile. It turned out
to be way less invasive than I expected too.
(I intentionally didn't add any way for clients to annotate each work-item
in a batch with their own private data. I think that'd be really useful and
would make implementing clients easier, but should be a separate patch).
This should be very useful for optimising FDWs, Postgres-XC, etc.
[1]
http://www.postgresql.org/message-id/CAMsr+YFgDUiJ37DEfPRk8WDBuZ58psdAYJd8iNFSaGxtw=wU3g@mail.gmail....
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
0001-Pipelining-batch-support-for-libpq.patch (text/x-patch, 78.0K) download
From 711a6951f57d84309bd2d7477a145ee3412b7967 Mon Sep 17 00:00:00 2001
From: Craig Ringer <craig@2ndquadrant.com>
Date: Fri, 20 May 2016 12:45:18 +0800
Subject: [PATCH] Pipelining (batch) support for libpq
Allow libpq clients to avoid excessive round trips by pipelining multiple
commands into batches. A sync is only sent at the end of a batch. Commands
in a batch succeed or fail together.
Includes a test program in src/test/examples/testlibpqbatch.c
---
doc/src/sgml/libpq.sgml | 478 +++++++++++++
src/interfaces/libpq/exports.txt | 7 +
src/interfaces/libpq/fe-connect.c | 16 +
src/interfaces/libpq/fe-exec.c | 572 +++++++++++++++-
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 17 +-
src/interfaces/libpq/libpq-fe.h | 13 +-
src/interfaces/libpq/libpq-int.h | 37 +-
src/test/examples/.gitignore | 1 +
src/test/examples/Makefile | 2 +-
src/test/examples/testlibpqbatch.c | 1254 +++++++++++++++++++++++++++++++++++
11 files changed, 2360 insertions(+), 43 deletions(-)
create mode 100644 src/test/examples/testlibpqbatch.c
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 3829a14..d6e896f 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4568,6 +4568,484 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up mulitiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/examples/libpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It somewhat increased client application
+ complexity and extra caution is required to prevent client/server network
+ deadlocks, but can offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ ("ping time") is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 9.6, but clients using it can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-pqbeginbatchmode"><function>PQbeginBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-pqisinbatchmode"><function>PQisInBatchMode(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not allowed. (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQsendEndBatch</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQgetNextQuery</function> to get results. It may eventually exit
+ batch mode with <function>PQendBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it's already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions like
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ etc. The asynchronous requests are followed by a <link
+ linkend="libpq-pqsendendbatch"><function>PQsendEndBatch(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server usually begins executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-pqgetnextquery"><function>PQgetNextQuery</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQgetNextQuery</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQsendEndBatch</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQsendEndBatch</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetNextQuery(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/examples/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-pqendbatchmode"><function>PQendBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-pqisinbatchmode">
+ <term>
+ <function>PQisInBatchMode</function>
+ <indexterm>
+ <primary>PQisInBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns 1 if a <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>, otherwise 0.
+
+<synopsis>
+int PQisInBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqbeginbatchmode">
+ <term>
+ <function>PQbeginBatchMode</function>
+ <indexterm>
+ <primary>PQbeginBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode and returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+<synopsis>
+int PQbeginBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqendbatchmode">
+ <term>
+ <function>PQendBatchMode</function>
+ <indexterm>
+ <primary>PQendBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results and returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQgetNextQuery</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+<synopsis>
+int PQendBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqsendendbatch">
+ <term>
+ <function>PQsendEndBatch</function>
+ <indexterm>
+ <primary>PQsendEndBatch</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQsendEndBatch(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqgetnextquery">
+ <term>
+ <function>PQgetNextQuery</function>
+ <indexterm>
+ <primary>PQgetNextQuery</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results. Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. See <link
+ linkend="libpq-batch-results">processing results</link>.
+
+<synopsis>
+int PQgetNextQuery(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqqueriesinbatch">
+ <term>
+ <function>PQqueriesInBatch</function>
+ <indexterm>
+ <primary>PQqueriesInBatch</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processsed.
+ This is the number of times <function>PQgetNextQuery</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQqueriesInBatch(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqbatchisaborted">
+ <term>
+ <function>PQbatchIsAborted</function>
+ <indexterm>
+ <primary>PQbatchIsAborted</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns 1 if the batch curently being received on a
+ <application>libpq</application> connection in <link
+ linkend="libpq-batch-mode">batch mode</link> is
+ <link linkend="libpq-batch-errors">aborted</link>, 0
+ otherwise. The aborted flag is cleared as soon as the result of the
+ <function>PQsendEndBatch</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function as they can tell
+ that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+
+<synopsis>
+int PQbatchIsAborted(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..1b0b8c5 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,10 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQisInBatchMode 172
+PQqueriesInBatch 173
+PQbeginBatchMode 174
+PQendBatchMode 175
+PQsendEndBatch 176
+PQgetNextQuery 177
+PQbatchIsAborted 178
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 9b2839b..78154e2 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -2949,6 +2949,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
/*
@@ -2995,6 +2996,21 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+ queue = conn->cmd_queue_recycle;
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 2621767..319c63e 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry* PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
/* ----------------
@@ -1090,7 +1095,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1113,6 +1118,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->in_batch)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1211,9 +1223,29 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
/* check the arguments */
if (!stmtName)
{
@@ -1269,18 +1301,21 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1290,10 +1325,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1340,6 +1379,81 @@ PQsendQueryPrepared(PGconn *conn,
resultFormat);
}
+/* Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry*
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry*) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/* Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/* Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/* Set up for processing a new query's results */
+static void
+PQstartProcessingNewQuery(PGconn *conn)
+{
+ /* initialize async result-accumulation state */
+ conn->result = NULL;
+ conn->next_result = NULL;
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+}
+
/*
* Common startup code for PQsendQuery and sibling routines
*/
@@ -1359,20 +1473,52 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && !conn->in_batch)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- conn->result = NULL;
- conn->next_result = NULL;
-
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ if (conn->in_batch)
+ {
+ /* When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQgetNextQuery(...) advances
+ * to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ case PGASYNC_IDLE:
+ fprintf(stderr, "internal error, idle state in batch mode");
+ abort();
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately */
+ PQstartProcessingNewQuery(conn);
+ }
/* ready to send command message */
return true;
@@ -1397,6 +1543,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1406,6 +1556,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1518,22 +1685,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1543,10 +1713,15 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
+
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1673,6 +1848,282 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/* PQisInBatchMode
+ * Return true if currently in batch mode
+ */
+int
+PQisInBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->in_batch;
+}
+
+/* PQqueriesInBatch
+ * Return true if there are queries currently pending in batch mode
+ */
+int
+PQqueriesInBatch(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->cmd_queue_head != NULL;
+}
+
+/* PQbatchIsAborted
+ * Batch being processed is aborted, results discarded until next sync
+ */
+int
+PQbatchIsAborted(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->batch_aborted;
+}
+
+/* Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * COPY is not permitted in batch mode.
+ *
+ * A set of commands is terminated by a PQsendEndBatch. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQendBatchMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbeginBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->in_batch)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->in_batch = true;
+ conn->batch_aborted = false;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/* End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQendBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (!conn->in_batch)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ fprintf(stderr, "internal error, IDLE in batch mode");
+ abort();
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ fprintf(stderr, "internal error, COPY in batch mode");
+ abort();
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->in_batch = false;
+ conn->batch_aborted = false;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/* End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQsendEndBatch command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQsendEndBatch(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (!conn->in_batch)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ fprintf(stderr, "internal error, IDLE in batch mode");
+ abort();
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ fprintf(stderr, "internal error, COPY in batch mode");
+ abort();
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/* PQgetNextQuery
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQgetNextQuery(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return FALSE;
+
+ if (!conn->in_batch)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ fprintf(stderr, "internal error, COPY in batch mode");
+ abort();
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ fprintf(stderr, "internal error, idle in batch mode");
+ abort();
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /* In batch mode but nothing left on the queue; caller can submit
+ * more work or PQendBatchMode() now. */
+ return false;
+ }
+
+ /* Pop the next query from the queue and set up the connection state
+ * as if it'd just been dispatched from a non-batched call */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ PQstartProcessingNewQuery(conn);
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_aborted && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+
+ /* Parse any available data */
+ parseInput(conn);
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1680,7 +2131,6 @@ PQisBusy(PGconn *conn)
* query work remains or an error has occurred (e.g. out of
* memory).
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1732,10 +2182,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->in_batch)
+ {
+ /* batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could still
+ * have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1915,6 +2386,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQexec in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2109,6 +2587,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2124,6 +2605,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2132,15 +2627,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && !conn->in_batch)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2154,10 +2652,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index f1b90f3..e824f36 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 0b8c62f..3d6e6c9 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->in_batch)
+ {
+ conn->batch_aborted = false;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -305,7 +313,7 @@ pqParseInput3(PGconn *conn)
* parsing until the application accepts the current
* result.
*/
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
return;
}
break;
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->in_batch)
+ conn->batch_aborted = true;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 9ca0756..a05298f 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -91,7 +91,9 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort earlier in a batch */
} ExecStatusType;
typedef enum
@@ -421,6 +423,15 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQisInBatchMode(PGconn *conn);
+extern int PQbatchIsAborted(PGconn *conn);
+extern int PQqueriesInBatch(PGconn *conn);
+extern int PQbeginBatchMode(PGconn *conn);
+extern int PQendBatchMode(PGconn *conn);
+extern int PQsendEndBatch(PGconn *conn);
+extern int PQgetNextQuery(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1183323..a21e07c 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,13 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch result,
+ More results expected from this query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -292,6 +296,22 @@ typedef struct pgDataValue
const char *value; /* data value, without zero-termination */
} PGdataValue;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* PGconn stores all the state data associated with a single connection
* to a backend.
@@ -356,6 +376,8 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ bool in_batch; /* connection is in batch (pipelined) mode */
+ bool batch_aborted; /* current batch is aborted, discarding until next Sync */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -363,6 +385,15 @@ struct pg_conn
PGnotify *notifyHead; /* oldest unreported Notify msg */
PGnotify *notifyTail; /* newest unreported Notify msg */
+ /* The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
/* See PQconnectPoll() for how we use 'int' and not 'pgsocket'. */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
diff --git a/src/test/examples/.gitignore b/src/test/examples/.gitignore
index 1957ec1..7a72420 100644
--- a/src/test/examples/.gitignore
+++ b/src/test/examples/.gitignore
@@ -4,3 +4,4 @@
/testlibpq4
/testlo
/testlo64
+/testlibpqbatch
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index 31da210..92a6faf 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
override LDLIBS := $(libpq_pgport) $(LDLIBS)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64 testlibpqbatch
all: $(PROGS)
diff --git a/src/test/examples/testlibpqbatch.c b/src/test/examples/testlibpqbatch.c
new file mode 100644
index 0000000..b137d6f
--- /dev/null
+++ b/src/test/examples/testlibpqbatch.c
@@ -0,0 +1,1254 @@
+/*
+ * src/test/examples/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution funtionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+
+#ifndef VERBOSE
+#define VERBOSE 0
+#endif
+
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQgetNextQuery when there might still be pending results */
+ if (PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (!PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (!PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent BEGIN\n");
+#endif
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent DROP\n");
+#endif
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent CREATE\n");
+#endif
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent PREPARE\n");
+#endif
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "next query!\n");
+#endif
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+#if VERBOSE
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+#endif
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+#endif
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+#if VERBOSE
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+#endif
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+#if VERBOSE
+ fprintf(stdout, "sent COMMIT\n");
+#endif
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendEndBatch(conn))
+ {
+#if VERBOSE
+ fprintf(stdout, "Dispatched end batch message\n");
+#endif
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows]]\n", progname);
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+ int number_of_rows = 10000;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 3)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ test_disallowed_in_batch(conn);
+ simple_batch(conn);
+ multi_batch(conn);
+ test_batch_abort(conn);
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.5.5
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-05-23 15:50 ` Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2 siblings, 1 reply; 182+ messages in thread
From: Andres Freund @ 2016-05-23 15:50 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: pgsql-hackers; Michael Paquier <michael.paquier@gmail.com>; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
Hi,
On 2016-05-23 17:19:09 +0800, Craig Ringer wrote:
> Hi all
> Following on from the foreign table batch inserts thread[1], here's a patch
> to add support for pipelining queries into asynchronous batches in libpq.
Yay!
> I'm measuring 300x (not %) performance improvements doing batches on
> servers over the Internet, so this seems pretty worthwhile. It turned out
> to be way less invasive than I expected too.
yay^2.
> (I intentionally didn't add any way for clients to annotate each work-item
> in a batch with their own private data. I think that'd be really useful and
> would make implementing clients easier, but should be a separate patch).
>
> This should be very useful for optimising FDWs, Postgres-XC, etc.
And optimizing normal clients.
Not easy, but I'd be very curious how much psql's performance improves
when replaying a .sql style dump, and replaying from a !tty fd, after
hacking it up to use the batch API.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2016-05-23 16:00 ` Michael Paquier <michael.paquier@gmail.com>
2016-05-24 01:20 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-24 04:02 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
0 siblings, 3 replies; 182+ messages in thread
From: Michael Paquier @ 2016-05-23 16:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Craig Ringer <craig@2ndquadrant.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On Mon, May 23, 2016 at 8:50 AM, Andres Freund <andres@anarazel.de> wrote:
> On 2016-05-23 17:19:09 +0800, Craig Ringer wrote:
>> Following on from the foreign table batch inserts thread[1], here's a patch
>> to add support for pipelining queries into asynchronous batches in libpq.
>
> Yay!
>> I'm measuring 300x (not %) performance improvements doing batches on
>> servers over the Internet, so this seems pretty worthwhile. It turned out
>> to be way less invasive than I expected too.
>
> yay^2.
I'll follow this mood. Yeha.
>> (I intentionally didn't add any way for clients to annotate each work-item
>> in a batch with their own private data. I think that'd be really useful and
>> would make implementing clients easier, but should be a separate patch).
>>
>> This should be very useful for optimising FDWs, Postgres-XC, etc.
>
> And optimizing normal clients.
>
> Not easy, but I'd be very curious how much psql's performance improves
> when replaying a .sql style dump, and replaying from a !tty fd, after
> hacking it up to use the batch API.
Did you consider the use of simple_list.c instead of introducing a new
mimic as PGcommandQueueEntry? It would be cool avoiding adding new
list emulations on frontends.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2016-05-24 01:20 ` Craig Ringer <craig@2ndquadrant.com>
2 siblings, 0 replies; 182+ messages in thread
From: Craig Ringer @ 2016-05-24 01:20 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 24 May 2016 at 00:00, Michael Paquier <michael.paquier@gmail.com> wrote:
> On Mon, May 23, 2016 at 8:50 AM, Andres Freund <andres@anarazel.de>
> wrote:
>
>> This should be very useful for optimising FDWs, Postgres-XC, etc.
> >
> > And optimizing normal clients.
> >
> > Not easy, but I'd be very curious how much psql's performance improves
> > when replaying a .sql style dump, and replaying from a !tty fd, after
> > hacking it up to use the batch API.
>
I didn't, but agree it'd be interesting. So would pg_restore, for that
matter, though its use of COPY for the bulk of its work means it wouldn't
make tons of difference.
I think it'd be safe to enable it automatically in psql's
--single-transaction mode. It's also safe to send anything after an
explicit BEGIN and until the next COMMIT as a batch from libpq, and since
it parses the SQL enough to detect statement boundaries already that
shouldn't be too hard to handle.
However: psql is synchronous, using the PQexec family of blocking calls.
It's all fairly well contained in SendQuery and PSQLexec, but making it use
batching still require restructuring those to use the asynchronous
nonblocking API and append the query to a pending-list, plus the addition
of a select() loop to handle results and dispatch more work. MainLoop()
isn't structured around a select or poll, it loops over gets. So while it'd
be interesting to see what difference batching made the changes to make
psql use it would be a bit more invasive. Far from a rewrite, but to avoid
lots of code duplication it'd have to change everything to use nonblocking
mode and a select loop, which is a big change for such a core tool.
This is a bit of a side-project and I've got to get back to "real work" so
I don't expect to do a proper patch for psql any time soon. I'd rather not
try to build too much on this until it's seen some review and I know the
API won't need a drastic rewrite anyway. I'll see if I can do a hacked-up
version one evening to see what it does for performance though.
Did you consider the use of simple_list.c instead of introducing a new
> mimic as PGcommandQueueEntry? It would be cool avoiding adding new
> list emulations on frontends.
>
Nope. I didn't realise it was there; I've done very little on the C client
and library side so far. So thanks, I'll update it accordingly.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2016-05-24 02:01 ` Craig Ringer <craig@2ndquadrant.com>
2016-05-27 02:43 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2 siblings, 2 replies; 182+ messages in thread
From: Craig Ringer @ 2016-05-24 02:01 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 24 May 2016 at 00:00, Michael Paquier <michael.paquier@gmail.com> wrote:
> On Mon, May 23, 2016 at 8:50 AM, Andres Freund <andres@anarazel.de> wrote:
>
> > yay^2.
>
> I'll follow this mood. Yeha.
BTW, I've publushed the HTML-ified SGML docs to
http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as a preview.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-05-27 02:43 ` Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
1 sibling, 0 replies; 182+ messages in thread
From: Tsunakawa, Takayuki @ 2016-05-27 02:43 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; Michael Paquier <michael.paquier@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
From: pgsql-hackers-owner@postgresql.org [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Craig Ringer
I'll follow this mood. Yeha.
BTW, I've publushed the HTML-ified SGML docs to http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as a preview.
Sorry for my late reply. Fantastic performance improvement, nice documentation, and amazing rapid development! I think I’ll join the review & testing in 2016/9 CF.
Regards
Takayuki Tsunakawa
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-06-03 11:51 ` Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
1 sibling, 1 reply; 182+ messages in thread
From: Dmitry Igrishin @ 2016-06-03 11:51 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
2016-05-24 5:01 GMT+03:00 Craig Ringer <craig@2ndquadrant.com>:
>
> On 24 May 2016 at 00:00, Michael Paquier <michael.paquier@gmail.com> wrote:
>>
>> On Mon, May 23, 2016 at 8:50 AM, Andres Freund <andres@anarazel.de> wrote:
>
>
>>
>> > yay^2.
>>
>> I'll follow this mood. Yeha.
>
>
>
> BTW, I've publushed the HTML-ified SGML docs to
> http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as a preview.
Typo detected: "Returns 1 if the batch curently being received" -- "curently".
--
// Dmitry.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
@ 2016-08-10 06:44 ` Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Michael Paquier @ 2016-08-10 06:44 UTC (permalink / raw)
To: Dmitry Igrishin <dmitigr@gmail.com>; +Cc: Craig Ringer <craig@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On Fri, Jun 3, 2016 at 8:51 PM, Dmitry Igrishin <dmitigr@gmail.com> wrote:
>> BTW, I've publushed the HTML-ified SGML docs to
>> http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as a preview.
> Typo detected: "Returns 1 if the batch curently being received" -- "curently".
I am looking a bit more seriously at this patch and assigned myself as
a reviewer.
testlibpqbatch.c:1239:73: warning: format specifies type 'long' but
the argument has type '__darwin_suseconds_t' (aka 'int') [-Wformat]
printf("batch insert elapsed: %ld.%06lds\n",
elapsed_time.tv_sec, elapsed_time.tv_usec);
macos complains here. You may want to replace %06lds by just %06d.
(lldb) bt
* thread #1: tid = 0x0000, 0x0000000108bcdd56
libpq.5.dylib`closePGconn(conn=0x00007fd642d002d0) + 438 at
fe-connect.c:3010, stop reason = signal SIGSTOP
* frame #0: 0x0000000108bcdd56
libpq.5.dylib`closePGconn(conn=0x00007fd642d002d0) + 438 at
fe-connect.c:3010
frame #1: 0x0000000108bc9db0
libpq.5.dylib`PQfinish(conn=0x00007fd642d002d0) + 32 at
fe-connect.c:3072
frame #2: 0x0000000108bc9ede
libpq.5.dylib`PQping(conninfo="dbname=postgres port=5432 host='/tmp'
connect_timeout=5") + 46 at fe-connect.c:539
frame #3: 0x0000000108bb5210
pg_ctl`test_postmaster_connection(pm_pid=78218, do_checkpoint='\0') +
976 at pg_ctl.c:681
frame #4: 0x0000000108bb388e pg_ctl`do_start + 302 at pg_ctl.c:915
frame #5: 0x0000000108bb29b4 pg_ctl`main(argc=5,
argv=0x00007fff5704e5c0) + 2836 at pg_ctl.c:2416
frame #6: 0x00007fff8b8b65ad libdyld.dylib`start + 1
(lldb) down 1
frame #0: 0x0000000108bcdd56
libpq.5.dylib`closePGconn(conn=0x00007fd642d002d0) + 438 at
fe-connect.c:3010
3007 queue = conn->cmd_queue_recycle;
3008 {
3009 PGcommandQueueEntry *prev = queue;
-> 3010 queue = queue->next;
3011 free(prev);
3012 }
3013 conn->cmd_queue_recycle = NULL;
This patch generates a core dump, use for example pg_ctl start -w and
you'll bump into the trace above. There is something wrong with the
queue handling.
Do you have plans for a more generic structure for the command queue list?
+ <application>libpq</application> supports queueing up mulitiple queries into
s/mulitiple/multiple/.
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/examples/libpqbatch.c</filename>.
+ </para>
You mean testlibpqbatch.c here.
+ <para>
+ Batching less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
"Batching *is* less useful".
src/test/examples/.gitignore needs a new entry for the new test binary.
+ fprintf(stderr, "internal error, COPY in batch mode");
+ abort();
I don't think that's a good idea. defaultNoticeProcessor can be
overridden to allow applications to have error messages sent
elsewhere. Error messages should also use libpq_gettext, and perhaps
be stored in conn->errorMessage as we do so for OOMs happening on
client-side and reporting them back even if they are not expected
(those are blocked PQsendQueryStart in your patch).
src/test/examples is a good idea to show people what this new API can
do, but this is never getting compiled. It could as well be possible
to include tests in src/test/modules/, in the same shape as what
postgres_fdw is doing by connecting to itself and link it to libpq. As
this patch complicates quote a lot fe-exec.c, I think that this would
be worth it. Thoughts?
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2016-08-23 00:27 ` Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2016-08-23 00:27 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 10 August 2016 at 14:44, Michael Paquier <michael.paquier@gmail.com>
wrote:
> On Fri, Jun 3, 2016 at 8:51 PM, Dmitry Igrishin <dmitigr@gmail.com> wrote:
> >> BTW, I've publushed the HTML-ified SGML docs to
> >> http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as a
> preview.
> > Typo detected: "Returns 1 if the batch curently being received" --
> "curently".
>
> I am looking a bit more seriously at this patch and assigned myself as
> a reviewer.
>
Much appreciated.
> testlibpqbatch.c:1239:73: warning: format specifies type 'long' but
> the argument has type '__darwin_suseconds_t' (aka 'int') [-Wformat]
> printf("batch insert elapsed: %ld.%06lds\n",
> elapsed_time.tv_sec, elapsed_time.tv_usec);
> macos complains here. You may want to replace %06lds by just %06d.
>
Yeah, or cast to a type known to be big enough. Will amend.
> This patch generates a core dump, use for example pg_ctl start -w and
> you'll bump into the trace above. There is something wrong with the
> queue handling.
>
Huh. I didn't see that here (Fedora 23). I'll look more closely.
> Do you have plans for a more generic structure for the command queue list?
>
No plans, no. This was a weekend experiment that turned into a useful patch
and I'm having to scrape up time for it amongst much more important things
like logical failover / sequence decoding and various other replication
work.
Thanks for the docs review too, will amend.
> + fprintf(stderr, "internal error, COPY in batch mode");
> + abort();
> I don't think that's a good idea. defaultNoticeProcessor can be
> overridden to allow applications to have error messages sent
> elsewhere. Error messages should also use libpq_gettext, and perhaps
> be stored in conn->errorMessage as we do so for OOMs happening on
> client-side and reporting them back even if they are not expected
> (those are blocked PQsendQueryStart in your patch).
>
> src/test/examples is a good idea to show people what this new API can
> do, but this is never getting compiled. It could as well be possible
> to include tests in src/test/modules/, in the same shape as what
> postgres_fdw is doing by connecting to itself and link it to libpq. As
> this patch complicates quote a lot fe-exec.c, I think that this would
> be worth it. Thoughts?
I didn't think it added much complexity to fe-exec.c personally. A lot of
the appeal is that it has very minor impact on anything that isn't using it.
I think it makes sense to (ab)use the recovery module tests for this,
invoking the test program from there.
Ideally I'd like to teach pgsql and pg_restore how to use async mode, but
that's a whole separate patch.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-08-23 06:56 ` Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
0 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2016-08-23 06:56 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 23 August 2016 at 08:27, Craig Ringer <craig@2ndquadrant.com> wrote:
> On 10 August 2016 at 14:44, Michael Paquier <michael.paquier@gmail.com>
> wrote:
>
>>
> I am looking a bit more seriously at this patch and assigned myself as
>> a reviewer.
>>
>
> Much appreciated.
>
>
>>
> macos complains here. You may want to replace %06lds by just %06d.
>>
>
> Yeah, or cast to a type known to be big enough. Will amend.
>
I used an upcast to (long), because on Linux it's a long. I don't see the
point of messing about adding a configure test for something this trivial.
> This patch generates a core dump, use for example pg_ctl start -w and
>> you'll bump into the trace above. There is something wrong with the
>> queue handling.
>>
>
> Huh. I didn't see that here (Fedora 23). I'll look more closely.
>
>
>> Do you have plans for a more generic structure for the command queue list?
>>
>
> No plans, no. This was a weekend experiment that turned into a useful
> patch and I'm having to scrape up time for it amongst much more important
> things like logical failover / sequence decoding and various other
> replication work.
>
> Thanks for the docs review too, will amend.
>
>
>> + fprintf(stderr, "internal error, COPY in batch mode");
>> + abort();
>> I don't think that's a good idea.
>
>
My thinking there was that it's a "shouldn't happen" case. It's a problem
in library logic. I'd use an Assert() here in the backend.
I could printfPQExpBuffer(...) an error and return failure instead if you
think that's more appropriate. I'm not sure how the app would handle it
correctly, but OTOH it's generally better for libraries not to call
abort(). So I'll do that, but since it's an internal error that's not meant
to happen I'll skip the gettext calls.
> Error messages should also use libpq_gettext, and perhaps
>> be stored in conn->errorMessage as we do so for OOMs happening on
>> client-side and reporting them back even if they are not expected
>> (those are blocked PQsendQueryStart in your patch).
>>
>
I didn't get that last part, re PQsendQueryStart.
> src/test/examples is a good idea to show people what this new API can
>> do, but this is never getting compiled. It could as well be possible
>> to include tests in src/test/modules/, in the same shape as what
>> postgres_fdw is doing by connecting to itself and link it to libpq. As
>> this patch complicates quote a lot fe-exec.c, I think that this would
>> be worth it. Thoughts?
>
>
I think it makes sense to use the TAP framework. Added
src/test/modules/test_libpq/ with a test for async mode. Others can be
added/migrated based on that. I thought it made more sense for the tests to
live there than in src/interfaces//libpq/ since they need test client
programs and shouldn't pollute the library directory.
I've made the docs changes too. Thanks.
I fixed the list handling error. I'm amazed it appears to run fine, and
without complaint from valgrind, here, since it was an accidentally
_deleted_ line.
Re lists, I looked at simple_list.c and it's exceedingly primitive. Using
it would mean more malloc()ing since we'll have a list cell and then a
struct pointed to it, and won't recycle members, but... whatever. It's not
going to matter a great deal. The reason I did it with an embedded list
originally was because that's how it's done for PGnotify, but that's not
exactly new code
The bigger problem is that simple_list also uses pg_malloc, which won't set
conn->errorMessage, it'll just fprintf() and exit(1). I'm not convinced
it's appropriate to use that for libpq.
For now I've left list handling unchanged. If it's to move to a generic
list, it should probably be one that knows how to use
pg_malloc_extended(size, MCXT_ALLOC_NO_OOM) and emit its own
libpq-error-handling-aware error. I'm not sure whether that list should use
cell heads embedded in the structures it manages or pointing to them,
either.
Updated patch attached.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
0001-Pipelining-batch-support-for-libpq-v2.patch (text/x-patch, 85.6K) download
From fad0def5570907f5c8e3b6d65d57e5e7678e7383 Mon Sep 17 00:00:00 2001
From: Craig Ringer <craig@2ndquadrant.com>
Date: Fri, 20 May 2016 12:45:18 +0800
Subject: [PATCH] Pipelining (batch) support for libpq
Allow libpq clients to avoid excessive round trips by pipelining multiple
commands into batches. A sync is only sent at the end of a batch. Commands
in a batch succeed or fail together.
Adds TAP tests for libpq at src/test/modules/test_libpq .
Includes a test program in src/test/modules/test_libpq/testlibpqbatch.c
---
doc/src/sgml/libpq.sgml | 478 ++++++++
src/interfaces/libpq/.gitignore | 1 +
src/interfaces/libpq/Makefile | 5 +
src/interfaces/libpq/exports.txt | 7 +
src/interfaces/libpq/fe-connect.c | 17 +
src/interfaces/libpq/fe-exec.c | 572 ++++++++-
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 17 +-
src/interfaces/libpq/libpq-fe.h | 13 +-
src/interfaces/libpq/libpq-int.h | 39 +-
src/interfaces/libpq/t/001_libpq_async.pl | 15 +
src/test/examples/.gitignore | 1 +
src/test/examples/Makefile | 2 +-
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 23 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1386 ++++++++++++++++++++++
18 files changed, 2571 insertions(+), 43 deletions(-)
create mode 100644 src/interfaces/libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index f22e3da..c565e18 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4568,6 +4568,484 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It somewhat increased client application
+ complexity and extra caution is required to prevent client/server network
+ deadlocks, but can offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ ("ping time") is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 9.6, but clients using it can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-pqbeginbatchmode"><function>PQbeginBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-pqisinbatchmode"><function>PQisInBatchMode(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not allowed. (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQsendEndBatch</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQgetNextQuery</function> to get results. It may eventually exit
+ batch mode with <function>PQendBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it's already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions like
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ etc. The asynchronous requests are followed by a <link
+ linkend="libpq-pqsendendbatch"><function>PQsendEndBatch(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server usually begins executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-pqgetnextquery"><function>PQgetNextQuery</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQgetNextQuery</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQsendEndBatch</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQsendEndBatch</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetNextQuery(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-pqendbatchmode"><function>PQendBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-pqisinbatchmode">
+ <term>
+ <function>PQisInBatchMode</function>
+ <indexterm>
+ <primary>PQisInBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns 1 if a <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>, otherwise 0.
+
+<synopsis>
+int PQisInBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqbeginbatchmode">
+ <term>
+ <function>PQbeginBatchMode</function>
+ <indexterm>
+ <primary>PQbeginBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode and returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+<synopsis>
+int PQbeginBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqendbatchmode">
+ <term>
+ <function>PQendBatchMode</function>
+ <indexterm>
+ <primary>PQendBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results and returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQgetNextQuery</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+<synopsis>
+int PQendBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqsendendbatch">
+ <term>
+ <function>PQsendEndBatch</function>
+ <indexterm>
+ <primary>PQsendEndBatch</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQsendEndBatch(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqgetnextquery">
+ <term>
+ <function>PQgetNextQuery</function>
+ <indexterm>
+ <primary>PQgetNextQuery</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results. Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. See <link
+ linkend="libpq-batch-results">processing results</link>.
+
+<synopsis>
+int PQgetNextQuery(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqqueriesinbatch">
+ <term>
+ <function>PQqueriesInBatch</function>
+ <indexterm>
+ <primary>PQqueriesInBatch</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processsed.
+ This is the number of times <function>PQgetNextQuery</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQqueriesInBatch(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqbatchisaborted">
+ <term>
+ <function>PQbatchIsAborted</function>
+ <indexterm>
+ <primary>PQbatchIsAborted</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns 1 if the batch curently being received on a
+ <application>libpq</application> connection in <link
+ linkend="libpq-batch-mode">batch mode</link> is
+ <link linkend="libpq-batch-errors">aborted</link>, 0
+ otherwise. The aborted flag is cleared as soon as the result of the
+ <function>PQsendEndBatch</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function as they can tell
+ that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+
+<synopsis>
+int PQbatchIsAborted(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/.gitignore b/src/interfaces/libpq/.gitignore
index cb96af7..4c0d934 100644
--- a/src/interfaces/libpq/.gitignore
+++ b/src/interfaces/libpq/.gitignore
@@ -22,3 +22,4 @@
/encnames.c
/wchar.c
/libpq.rc
+/tmp_check
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 0b4065e..573b265 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -129,6 +129,11 @@ install: all installdirs install-lib
installcheck:
$(MAKE) -C test $@
+check: prove-check
+
+prove-check:
+ $(prove_check)
+
installdirs: installdirs-lib
$(MKDIR_P) '$(DESTDIR)$(includedir)' '$(DESTDIR)$(includedir_internal)' '$(DESTDIR)$(datadir)'
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..1b0b8c5 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,10 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQisInBatchMode 172
+PQqueriesInBatch 173
+PQbeginBatchMode 174
+PQendBatchMode 175
+PQsendEndBatch 176
+PQgetNextQuery 177
+PQbatchIsAborted 178
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 76b61bd..8725dc6 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -2949,6 +2949,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
/*
@@ -2995,6 +2996,22 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+ queue = conn->cmd_queue_recycle;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index d1b91c8..023edab 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry* PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
/* ----------------
@@ -1107,7 +1112,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1130,6 +1135,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->in_batch)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1228,9 +1240,29 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
/* check the arguments */
if (!stmtName)
{
@@ -1286,18 +1318,21 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1307,10 +1342,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1357,6 +1396,81 @@ PQsendQueryPrepared(PGconn *conn,
resultFormat);
}
+/* Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry*
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry*) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/* Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/* Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/* Set up for processing a new query's results */
+static void
+PQstartProcessingNewQuery(PGconn *conn)
+{
+ /* initialize async result-accumulation state */
+ conn->result = NULL;
+ conn->next_result = NULL;
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+}
+
/*
* Common startup code for PQsendQuery and sibling routines
*/
@@ -1376,20 +1490,52 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && !conn->in_batch)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- conn->result = NULL;
- conn->next_result = NULL;
-
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ if (conn->in_batch)
+ {
+ /* When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQgetNextQuery(...) advances
+ * to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately */
+ PQstartProcessingNewQuery(conn);
+ }
/* ready to send command message */
return true;
@@ -1414,6 +1560,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1573,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1702,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1730,15 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
+
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1865,282 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/* PQisInBatchMode
+ * Return true if currently in batch mode
+ */
+int
+PQisInBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->in_batch;
+}
+
+/* PQqueriesInBatch
+ * Return true if there are queries currently pending in batch mode
+ */
+int
+PQqueriesInBatch(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->cmd_queue_head != NULL;
+}
+
+/* PQbatchIsAborted
+ * Batch being processed is aborted, results discarded until next sync
+ */
+int
+PQbatchIsAborted(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->batch_aborted;
+}
+
+/* Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * COPY is not permitted in batch mode.
+ *
+ * A set of commands is terminated by a PQsendEndBatch. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQendBatchMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbeginBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->in_batch)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->in_batch = true;
+ conn->batch_aborted = false;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/* End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQendBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (!conn->in_batch)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->in_batch = false;
+ conn->batch_aborted = false;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/* End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQsendEndBatch command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQsendEndBatch(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (!conn->in_batch)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/* PQgetNextQuery
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQgetNextQuery(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return FALSE;
+
+ if (!conn->in_batch)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /* In batch mode but nothing left on the queue; caller can submit
+ * more work or PQendBatchMode() now. */
+ return false;
+ }
+
+ /* Pop the next query from the queue and set up the connection state
+ * as if it'd just been dispatched from a non-batched call */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ PQstartProcessingNewQuery(conn);
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_aborted && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+
+ /* Parse any available data */
+ parseInput(conn);
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1697,7 +2148,6 @@ PQisBusy(PGconn *conn)
* query work remains or an error has occurred (e.g. out of
* memory).
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1749,10 +2199,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->in_batch)
+ {
+ /* batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could still
+ * have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2403,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQexec in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2604,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2622,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2644,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && !conn->in_batch)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2669,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index f1b90f3..e824f36 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index a8a987a..78ffc1e 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->in_batch)
+ {
+ conn->batch_aborted = false;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -305,7 +313,7 @@ pqParseInput3(PGconn *conn)
* parsing until the application accepts the current
* result.
*/
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
return;
}
break;
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->in_batch)
+ conn->batch_aborted = true;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 9ca0756..a05298f 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -91,7 +91,9 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort earlier in a batch */
} ExecStatusType;
typedef enum
@@ -421,6 +423,15 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQisInBatchMode(PGconn *conn);
+extern int PQbatchIsAborted(PGconn *conn);
+extern int PQqueriesInBatch(PGconn *conn);
+extern int PQbeginBatchMode(PGconn *conn);
+extern int PQendBatchMode(PGconn *conn);
+extern int PQsendEndBatch(PGconn *conn);
+extern int PQgetNextQuery(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1183323..0954239 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,13 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch result,
+ More results expected from this query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -292,6 +296,22 @@ typedef struct pgDataValue
const char *value; /* data value, without zero-termination */
} PGdataValue;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* PGconn stores all the state data associated with a single connection
* to a backend.
@@ -356,6 +376,8 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ bool in_batch; /* connection is in batch (pipelined) mode */
+ bool batch_aborted; /* current batch is aborted, discarding until next Sync */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -363,6 +385,15 @@ struct pg_conn
PGnotify *notifyHead; /* oldest unreported Notify msg */
PGnotify *notifyTail; /* newest unreported Notify msg */
+ /* The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
/* See PQconnectPoll() for how we use 'int' and not 'pgsocket'. */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
@@ -658,6 +689,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
diff --git a/src/interfaces/libpq/t/001_libpq_async.pl b/src/interfaces/libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..d8f153e
--- /dev/null
+++ b/src/interfaces/libpq/t/001_libpq_async.pl
@@ -0,0 +1,15 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+$node->stop('fast');
diff --git a/src/test/examples/.gitignore b/src/test/examples/.gitignore
index 1957ec1..7a72420 100644
--- a/src/test/examples/.gitignore
+++ b/src/test/examples/.gitignore
@@ -4,3 +4,4 @@
/testlibpq4
/testlo
/testlo64
+/testlibpqbatch
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index 31da210..92a6faf 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
override LDLIBS := $(libpq_pgport) $(LDLIBS)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64 testlibpqbatch
all: $(PROGS)
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..7f1b8b0
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check: prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..28c3eb5
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 5;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..2dc485d
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1386 @@
+/*
+ * src/test/examples/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution funtionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+
+#ifndef VERBOSE
+#define VERBOSE 0
+#endif
+
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQgetNextQuery when there might still be pending results */
+ if (PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (!PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (!PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent BEGIN\n");
+#endif
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent DROP\n");
+#endif
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent CREATE\n");
+#endif
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent PREPARE\n");
+#endif
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "next query!\n");
+#endif
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+#if VERBOSE
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+#endif
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+#endif
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+#if VERBOSE
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+#endif
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+#if VERBOSE
+ fprintf(stdout, "sent COMMIT\n");
+#endif
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendEndBatch(conn))
+ {
+#if VERBOSE
+ fprintf(stdout, "Dispatched end batch message\n");
+#endif
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[MAXINTLEN+2];
+ int formatted = snprintf(&buf[0], MAXINTLEN+1, "%d\n", nrows);
+ if (formatted >= MAXINTLEN+1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long)elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long)elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long)elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognised test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.5.5
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-09-06 08:10 ` Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2016-09-06 08:10 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
Craig Ringer wrote:
> Updated patch attached.
Please find attached a couple fixes for typos I've came across in
the doc part.
Also it appears that PQqueriesInBatch() doesn't work as documented.
It says:
"Returns the number of queries still in the queue for this batch"
but in fact it's implemented as a boolean:
+/* PQqueriesInBatch
+ * Return true if there are queries currently pending in batch mode
+ */+int
+PQqueriesInBatch(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->cmd_queue_head != NULL;
+}
However, is this function really needed? It doesn't seem essential to
the API. You don't call it in the test program either.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
diff-typos.txt (text/plain, 1.9K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 73c6c03..af4f922 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4653,7 +4653,7 @@ int PQflush(PGconn *conn);
</programlisting>
could be much more efficiently done with:
<programlisting>
- UPDATE mytable SET x = x + 1;
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
</programlisting>
</para>
@@ -4696,7 +4696,7 @@ int PQflush(PGconn *conn);
<link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
blocking mode it is possible for a client/server deadlock to occur. The
client will block trying to send queries to the server, but the server will
- block trying to send results from queries it's already processed to the
+ block trying to send results from queries it has already processed to the
client. This only occurs when the client sends enough queries to fill its
output buffer and the server's receive buffer before switching to
processing input from the server, but it's hard to predict exactly when
@@ -5015,7 +5015,7 @@ int PQgetNextQuery(PGconn *conn);
<listitem>
<para>
Returns the number of queries still in the queue for this batch, not
- including any query that's currently having results being processsed.
+ including any query that's currently having results being processed.
This is the number of times <function>PQgetNextQuery</function> has to be
called before the query queue is empty again.
@@ -5037,7 +5037,7 @@ int PQqueriesInBatch(PGconn *conn);
<listitem>
<para>
- Returns 1 if the batch curently being received on a
+ Returns 1 if the batch currently being received on a
<application>libpq</application> connection in <link
linkend="libpq-batch-mode">batch mode</link> is
<link linkend="libpq-batch-errors">aborted</link>, 0
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2016-09-06 11:01 ` Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2016-09-06 11:01 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 6 September 2016 at 16:10, Daniel Verite <daniel@manitou-mail.org> wrote:
> Craig Ringer wrote:
>
>> Updated patch attached.
>
> Please find attached a couple fixes for typos I've came across in
> the doc part.
Thanks, will apply and post a rebased patch soon, or if someone picks
this up in the mean time they can apply your diff on top of the patch.
> Also it appears that PQqueriesInBatch() doesn't work as documented.
> It says:
> "Returns the number of queries still in the queue for this batch"
> but in fact it's implemented as a boolean:
Whoops. Will fix.
I think the function is useful and necessary. There's no reason not to
expose that, but also it's good for when your query dispatch isn't as
tightly coupled to your query handling as in the example, so your app
might need to keep processing until it sees the end of queued results.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-10-03 02:10 ` Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Michael Paquier @ 2016-10-03 02:10 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On Tue, Sep 6, 2016 at 8:01 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
> On 6 September 2016 at 16:10, Daniel Verite <daniel@manitou-mail.org> wrote:
>> Craig Ringer wrote:
>>
>>> Updated patch attached.
>>
>> Please find attached a couple fixes for typos I've came across in
>> the doc part.
>
> Thanks, will apply and post a rebased patch soon, or if someone picks
> this up in the mean time they can apply your diff on top of the patch.
Could you send an updated patch then? At the same time I am noticing
that git --check is complaining... This patch has tests and a
well-documented feature, so I'll take a look at it soon at the top of
my list. Moved to next CF for now.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2016-10-03 03:48 ` Craig Ringer <craig@2ndquadrant.com>
2016-10-03 14:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-10-05 04:54 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
0 siblings, 3 replies; 182+ messages in thread
From: Craig Ringer @ 2016-10-03 03:48 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 3 October 2016 at 10:10, Michael Paquier <michael.paquier@gmail.com> wrote:
> On Tue, Sep 6, 2016 at 8:01 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
>> On 6 September 2016 at 16:10, Daniel Verite <daniel@manitou-mail.org> wrote:
>>> Craig Ringer wrote:
>>>
>>>> Updated patch attached.
>>>
>>> Please find attached a couple fixes for typos I've came across in
>>> the doc part.
>>
>> Thanks, will apply and post a rebased patch soon, or if someone picks
>> this up in the mean time they can apply your diff on top of the patch.
>
> Could you send an updated patch then? At the same time I am noticing
> that git --check is complaining... This patch has tests and a
> well-documented feature, so I'll take a look at it soon at the top of
> my list. Moved to next CF for now.
Thanks.
I'd really like to teach psql in non-interactive mode to use it, but
(a) I'm concerned about possible subtle behaviour differences arising
if we do that and (b) I won't have the time. I think it's mostly of
interest to app authors and driver developers and that's what it's
aimed at. pg_restore could benefit a lot too.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-10-03 14:52 ` Daniel Verite <daniel@manitou-mail.org>
2016-10-04 07:15 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2 siblings, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2016-10-03 14:52 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
Craig Ringer wrote:
> I think it's mostly of interest to app authors and driver developers
> and that's what it's aimed at. pg_restore could benefit a lot too.
Wouldn't pgbench benefit from it?
It was mentioned some time ago [1], in relationship to the
\into construct, how client-server latency was important enough to
justify the use of a "\;" separator between statements, to send them
as a group.
But with the libpq batch API, maybe this could be modernized
with meta-commands like this:
\startbatch
...
\endbatch
which would essentially call PQbeginBatchMode() and PQsendEndBatch().
Inside the batch section, collecting results would be interleaved
with sending queries.
Interdepencies between results and subsequent queries could
be handled or ignored, depending on how sophisticated we'd want
this.
This might also draw more users to the batch API, because
it would make it easier to check how exactly it affects
the performance of specific sequences of SQL statements to be
grouped in a batch.
For instance it would make sense for programmers to benchmark mock-ups
of their code with pgbench with/without batching, before embarking on
adapting it from blocking mode to asynchronous/non-blocking mode.
[1]
https://www.postgresql.org/message-id/alpine.DEB.2.20.1607140925400.1962%40sto
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 14:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2016-10-04 07:15 ` Michael Paquier <michael.paquier@gmail.com>
2016-10-04 08:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-04 08:39 ` Re: PATCH: Batch/pipelining support for libpq Gavin Flower <GavinFlower@archidevsys.co.nz>
0 siblings, 2 replies; 182+ messages in thread
From: Michael Paquier @ 2016-10-04 07:15 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Craig Ringer <craig@2ndquadrant.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On Mon, Oct 3, 2016 at 11:52 PM, Daniel Verite <daniel@manitou-mail.org> wrote:
> Wouldn't pgbench benefit from it?
> It was mentioned some time ago [1], in relationship to the
> \into construct, how client-server latency was important enough to
> justify the use of a "\;" separator between statements, to send them
> as a group.
>
> But with the libpq batch API, maybe this could be modernized
> with meta-commands like this:
> \startbatch
> ...
> \endbatch
Or just \batch [on|off], which sounds like a damn good idea to me for
some users willing to test some workloads before integrating it in an
application.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 14:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-10-04 07:15 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2016-10-04 08:38 ` Craig Ringer <craig@2ndquadrant.com>
1 sibling, 0 replies; 182+ messages in thread
From: Craig Ringer @ 2016-10-04 08:38 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Manuel Kniep <m.kniep@web.de>; pgsql-hackers; Andres Freund <andres@anarazel.de>; Daniel Verite <daniel@manitou-mail.org>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Dmitry Igrishin <dmitigr@gmail.com>
On 4 Oct. 2016 15:15, "Michael Paquier" <michael.paquier@gmail.com> wrote:
>
> On Mon, Oct 3, 2016 at 11:52 PM, Daniel Verite <daniel@manitou-mail.org>
wrote:
> > Wouldn't pgbench benefit from it?
> > It was mentioned some time ago [1], in relationship to the
> > \into construct, how client-server latency was important enough to
> > justify the use of a "\;" separator between statements, to send them
> > as a group.
> >
> > But with the libpq batch API, maybe this could be modernized
> > with meta-commands like this:
> > \startbatch
> > ...
> > \endbatch
>
> Or just \batch [on|off], which sounds like a damn good idea to me for
> some users willing to test some workloads before integrating it in an
> application.
A batch jsnt necessarily terminated by a commit, so I'm more keen on
start/end batch. It's more in line with begin/commit. Batch is not only a
mode, you also have to delineate batches.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 14:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-10-04 07:15 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2016-10-04 08:39 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
1 sibling, 0 replies; 182+ messages in thread
From: Gavin Flower @ 2016-10-04 08:39 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; +Cc: Craig Ringer <craig@2ndquadrant.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 04/10/16 20:15, Michael Paquier wrote:
> On Mon, Oct 3, 2016 at 11:52 PM, Daniel Verite <daniel@manitou-mail.org> wrote:
>> Wouldn't pgbench benefit from it?
>> It was mentioned some time ago [1], in relationship to the
>> \into construct, how client-server latency was important enough to
>> justify the use of a "\;" separator between statements, to send them
>> as a group.
>>
>> But with the libpq batch API, maybe this could be modernized
>> with meta-commands like this:
>> \startbatch
>> ...
>> \endbatch
> Or just \batch [on|off], which sounds like a damn good idea to me for
> some users willing to test some workloads before integrating it in an
> application.
+1
'\batch' is a bit easier, to find, & to remember than '\startbatch'
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-10-05 04:54 ` Michael Paquier <michael.paquier@gmail.com>
2016-10-14 01:04 ` Re: PATCH: Batch/pipelining support for libpq Jim Nasby <Jim.Nasby@BlueTreble.com>
2 siblings, 1 reply; 182+ messages in thread
From: Michael Paquier @ 2016-10-05 04:54 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On Mon, Oct 3, 2016 at 12:48 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
> On 3 October 2016 at 10:10, Michael Paquier <michael.paquier@gmail.com> wrote:
>> On Tue, Sep 6, 2016 at 8:01 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
>>> On 6 September 2016 at 16:10, Daniel Verite <daniel@manitou-mail.org> wrote:
>>>> Craig Ringer wrote:
>>>>
>>>>> Updated patch attached.
>>>>
>>>> Please find attached a couple fixes for typos I've came across in
>>>> the doc part.
>>>
>>> Thanks, will apply and post a rebased patch soon, or if someone picks
>>> this up in the mean time they can apply your diff on top of the patch.
>>
>> Could you send an updated patch then? At the same time I am noticing
>> that git --check is complaining... This patch has tests and a
>> well-documented feature, so I'll take a look at it soon at the top of
>> my list. Moved to next CF for now.
>
> Thanks.
>
> I'd really like to teach psql in non-interactive mode to use it, but
> (a) I'm concerned about possible subtle behaviour differences arising
> if we do that and (b) I won't have the time. I think it's mostly of
> interest to app authors and driver developers and that's what it's
> aimed at. pg_restore could benefit a lot too.
Looking at it now... The most interesting comments are first.
I wanted to congratulate you. I barely see a patch with this level of
details done within the first versions. Anybody can review the patch
just by looking at the code and especially the docs without looking at
the thread. There are even tests to show what this does, for the
client.
+PQisInBatchMode 172
+PQqueriesInBatch 173
+PQbeginBatchMode 174
+PQendBatchMode 175
+PQsendEndBatch 176
+PQgetNextQuery 177
+PQbatchIsAborted 178
This set of routines is a bit inconsistent. Why not just prefixing
them with PQbatch? Like that for example:
PQbatchStatus(): consists of disabled/inactive/none, active, error.
This covers both PQbatchIsAborted() and PQisInBatchMode().
PQbatchBegin()
PQbatchEnd()
PQbatchQueueSync() or PQbatchQueueFlush, same as PQsendEndBatch() to
add and process a sync message into the queue.
PQbatchQueueCount(): returns N>0 if there are N entries, 0 if empty,
-1 on failure
PQbatchQueueProcess(): returns 1 if process can begin, 0 if not, -1 on
failure (OOM)
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It somewhat increased client application
+ complexity and extra caution is required to prevent client/server network
+ deadlocks, but can offer considerable performance improvements.
+ </para>
I would reword that a bit "it increases client application complexity
and extra caution is required to prevent client/server deadlocks but
offers considerable performance improvements".
+ ("ping time") is high, and when many small operations are being
performed in
Nit: should use <quote> here. Still not quoting it would be fine.
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions like
+ <function>PQsendQueryParams</function>,
<function>PQsendPrepare</function>,
+ etc. The asynchronous requests are followed by a <link
It may be better to list all the functions here, PQSendQuery
* query work remains or an error has occurred (e.g. out of
* memory).
*/
-
PGresult *
PQgetResult(PGconn *conn)
Some noise in the patch.
git diff --check complains:
usage_exit(argv[0]);
warning: 1 line adds whitespace errors.
+++ b/src/interfaces/libpq/t/001_libpq_async.pl
@@ -0,0 +1,15 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+$node->stop('fast');
Er... This does nothing..
The changes in src/test/examples/ are not necessary anymore. You moved
all the tests to test_libpq (for the best actually).
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
This should free prev->query.
/* blah comment
* blah2 comment
*/
A lot of comment blocks are like that, those should be reformated.
Running directly make check in src/test/modules/test_libpq/ does not work:
# Postmaster PID for node "main" is 10225
# Running: testlibpqbatch dbname=postgres 10000 disallowed_in_batch
Command 'testlibpqbatch' not found in [...PATH list ...]
The problem is that testlibpqbatch is not getting compiled but I think
it should.
+ * testlibpqbatch.c
+ * Test of batch execution funtionality
[...]
+ fprintf(stderr, "%s is not a recognised test name\n", argv[3]);
s/funtionality/functionality/
s/recognised/recognized/
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n",
PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n",
PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n",
PQerrorMessage(conn));
+ goto fail;
+ }
May be better to use a loop here and set in the queue a bunch of queries..
You could just remove the VERBOSE flag in the tests, having a test
more talkative is always better.
+ <para>
+ The batch API was introduced in PostgreSQL 9.6, but clients using it can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
Postgres 10, not 9.6.
It may be a good idea to check for PG_PROTOCOL_MAJOR < 3 and issue an
error for the new routines.
+int
+PQgetNextQuery(PGconn *conn)
[...]
+ return false;
The new routines had better return explicitly 0 or 1, not a boolean
for consistency with the rest.
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. See <link
It would be useful for the user to make a difference between those
different statuses. As of your patch, PQgetNextQuery() returns false/0
in all those cases so it is a bit hard to know what's going on. Maybe
PQgetNextQuery() could be renamed to PQbatchGetNext, PQbatchQueueNext
to outline the fact that it is beginning the next query in the queue.
This helps also to understand that this can just be used with the
batch mode. Actually no, I am wrong. It is possible to guess each one
of those errors with respectively PQgetResult == NULL,
PQisInBatchMode() and PQqueriesInBatch(). Definitely it should be
mentioned in the docs that it is possible to make a difference between
all those states.
+ entry = PQmakePipelinedCommand(conn);
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
PQmakePipelinedCommand() returns NULL, and boom.
+ bool in_batch; /* connection is in batch (pipelined) mode */
+ bool batch_aborted; /* current batch is aborted,
discarding until next Sync */
Having only one flag would be fine. batch_aborted is set of used only
when in_batch is used, so both have a strong link.
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
No, it is put in the queue.
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processsed.
+ This is the number of times <function>PQgetNextQuery</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQqueriesInBatch(PGconn *conn);
+</synopsis>
This is not true. It does not return a count, just 0 or 1.
It may be a good idea to add a test for COPY and trigger a failure.
If I read the code correctly, it seems to me that it is possible to
enable the batch mode, and then to use PQexec(), PQsendPrepare will
just happily process queue the command. Shouldn't PQexec() be
prevented in batch mode? Similar remark for PQexecParams(),
PQexecPrepared() PQdescribePrepared and PQprepare(). In short
everything calling PQexecFinish().
I haven't run the tests directly on Windows wiht MSVC, but they won't
run as vcregress modulescheck lacks knowledge about modules using TAP.
I cannot blame you for that..
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-05 04:54 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2016-10-14 01:04 ` Jim Nasby <Jim.Nasby@BlueTreble.com>
0 siblings, 0 replies; 182+ messages in thread
From: Jim Nasby @ 2016-10-14 01:04 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; Craig Ringer <craig@2ndquadrant.com>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 10/4/16 11:54 PM, Michael Paquier wrote:
> + <para>
> + Much like asynchronous query mode, there is no performance disadvantage to
> + using batching and pipelining. It somewhat increased client application
> + complexity and extra caution is required to prevent client/server network
> + deadlocks, but can offer considerable performance improvements.
> + </para>
> I would reword that a bit "it increases client application complexity
> and extra caution is required to prevent client/server deadlocks but
> offers considerable performance improvements".
Unrelated, but another doc bug, on line 4647:
+ The batch API was introduced in PostgreSQL 9.6, but clients
using it can
That should read 10.0 (or just 10?)
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532) mobile: 512-569-9461
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-11-18 06:04 ` Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2 siblings, 1 reply; 182+ messages in thread
From: Tsunakawa, Takayuki @ 2016-11-18 06:04 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; Michael Paquier <michael.paquier@gmail.com>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
Hello, Craig,
I'm sorry to be late to review your patch. I've just been able to read the HTML doc first. Can I get the latest .patch file for reading and running the code?
Here are some comments and questions. I tried to avoid the same point as other reviewers, but there may be an overlap.
(1)
The example
UPDATE mytable SET x = x + 1;
should be
UPDATE mytable SET x = x + 1 WHERE id = 42;
(2)
"The server usually begins executing the batch before all commands in the batch are queued and the end of batch command is sent."
Does this mean that the app developer cannot control or predict how many TCP transmissions a batch is sent with? For example, if I want to insert 10 rows into a table in bulk, can I send those 10 rows (and the end of batch command) efficiently in one TCP transmission, or are they split by libpq into multiple TCP transmissions?
(3)
"To avoid deadlocks on large batches the client should be structured around a nonblocking I/O loop using a function like select, poll, epoll, WaitForMultipleObjectEx, etc."
Can't we use some (new) platform-independent API instead of using poll() or WaitForMultipleObject()? e.g. some thin wrapper around pqWait(). It seems a bit burdonsome to have to use an OS-specific API to just wait for libpq. Apart from that, it does not seem possible to wait for the socket in 64-bit apps on Windows, because SOCKET is 64-bit while PQsocket() returns int.
[winsock2.h]
/*
* The new type to be used in all
* instances which refer to sockets.
*/
typedef UINT_PTR SOCKET;
Regards
Takayuki Tsunakawa
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
@ 2016-11-18 08:18 ` Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2016-11-18 08:18 UTC (permalink / raw)
To: Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 18 November 2016 at 14:04, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:
> Hello, Craig,
>
> I'm sorry to be late to review your patch. I've just been able to read the HTML doc first. Can I get the latest .patch file for reading and running the code?
The latest is what's attached upthread and what's in the git repo also
referenced upthread.
I haven't been able to update in response to more recent review due to
other development commitments. At this point I doubt I'll be able to
get update it again in time for v10, so anyone who wants to adopt it
is welcome.
> Here are some comments and questions. I tried to avoid the same point as other reviewers, but there may be an overlap.
>
>
> (1)
> The example
> UPDATE mytable SET x = x + 1;
> should be
> UPDATE mytable SET x = x + 1 WHERE id = 42;
Good catch.
> (2)
> "The server usually begins executing the batch before all commands in the batch are queued and the end of batch command is sent."
>
> Does this mean that the app developer cannot control or predict how many TCP transmissions a batch is sent with?
That's not what that sentence means since the TCP layer is much lower
level, but what you say is also true.
All the docs are saying there is that there's no explicit control over
when we start sending the batch to the server. How that translates to
individual TCP packets, etc, is not its problem.
> For example, if I want to insert 10 rows into a table in bulk, can I send those 10 rows (and the end of batch command) efficiently in one TCP transmission, or are they split by libpq into multiple TCP transmissions?
libpq neither knows nor cares about individual TCP packets. It sends
things to the kernel and lets the kernel deal with that.
That said, you can use socket options TCP_CORK and TCP_NODELAY to get
some control over how and when data is sent. If you know you're about
to send more, you might cork the socket to give the kernel a hint that
it should expect more data to send.
> (3)
> "To avoid deadlocks on large batches the client should be structured around a nonblocking I/O loop using a function like select, poll, epoll, WaitForMultipleObjectEx, etc."
>
> Can't we use some (new) platform-independent API instead of using poll() or WaitForMultipleObject()? e.g. some thin wrapper around pqWait(). It seems a bit burdonsome to have to use an OS-specific API to just wait for libpq. Apart from that, it does not seem possible to wait for the socket in 64-bit apps on Windows, because SOCKET is 64-bit while PQsocket() returns int.
IMO this problem is out of scope for this patch. A wait abstraction
might be nice, but next thing we know we'll be reinventing APR or
NSPR, I think that's a totally different problem.
Not being able to get a win32 SOCKET from libpq seems like a bit of an
oversight, and it'd definitely be good to address that to make async
mode more usable on Windows. There's some other ugliness in PQsocket
already per the comments there. I think it should be a separate patch,
though.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-11-22 07:14 ` Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Haribabu Kommi @ 2016-11-22 07:14 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Michael Paquier <michael.paquier@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On Fri, Nov 18, 2016 at 7:18 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
> The latest is what's attached upthread and what's in the git repo also
> referenced upthread.
>
> I haven't been able to update in response to more recent review due to
> other development commitments. At this point I doubt I'll be able to
> get update it again in time for v10, so anyone who wants to adopt it
> is welcome.
>
Currently patch status is marked as "returned with feedback" in the 11-2016
commitfest. Anyone who wants to work on it can submit the updated patch
by taking care of all review comments and change the status of the patch
at any time.
Thanks for the patch.
Regards,
Hari Babu
Fujitsu Australia
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
@ 2016-11-22 07:31 ` Craig Ringer <craig@2ndquadrant.com>
2017-02-14 07:27 ` Re: PATCH: Batch/pipelining support for libpq Iwata, Aya <iwata.aya@jp.fujitsu.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
0 siblings, 2 replies; 182+ messages in thread
From: Craig Ringer @ 2016-11-22 07:31 UTC (permalink / raw)
To: Haribabu Kommi <kommi.haribabu@gmail.com>; +Cc: Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Michael Paquier <michael.paquier@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 22 November 2016 at 15:14, Haribabu Kommi <kommi.haribabu@gmail.com> wrote:
>
> On Fri, Nov 18, 2016 at 7:18 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
>>
>> The latest is what's attached upthread and what's in the git repo also
>> referenced upthread.
>>
>> I haven't been able to update in response to more recent review due to
>> other development commitments. At this point I doubt I'll be able to
>> get update it again in time for v10, so anyone who wants to adopt it
>> is welcome.
>
>
> Currently patch status is marked as "returned with feedback" in the 11-2016
> commitfest. Anyone who wants to work on it can submit the updated patch
> by taking care of all review comments and change the status of the patch
> at any time.
>
> Thanks for the patch.
Thanks. Sorry I haven't had time to work on it. Priorities.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2017-02-14 07:27 ` Iwata, Aya <iwata.aya@jp.fujitsu.com>
1 sibling, 0 replies; 182+ messages in thread
From: Iwata, Aya @ 2017-02-14 07:27 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Michael Paquier <michael.paquier@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Haribabu Kommi <kommi.haribabu@gmail.com>; pgsql-hackers
Hi,
On 18 November 2016 at 08:18, Craig Ringer wrote:
>At this point I doubt I'll be able to
>get update it again in time for v10, so anyone who wants to adopt it
>is welcome.
I am interested in pipeline/batch support for ECPG, and found this thread.
I updated Craig's patch [1] to apply this one to HEAD. Moreover, I fixed an easy typo.
First, I'm changing PQqueriesInBatch() to work as documented.
After that, I plan to reflect contents of reviews in the patch.
On Mon, Oct 3, 2016 at 11:52 PM, Daniel Verite wrote:
> Wouldn't pgbench benefit from it?
> It was mentioned some time ago [1], in relationship to the
> \into construct, how client-server latency was important enough to
> justify the use of a "\;" separator between statements, to send them
> as a group.
>
> But with the libpq batch API, maybe this could be modernized
> with meta-commands like this:
> \startbatch
> ...
> \endbatch
I'm planning to work on meta-commands to support batch mode after committing this patch successfully.
[1]https://github.com/2ndQuadrant/postgres/tree/dev/libpq-async-batch
Regards,
Aya Iwata
FUJITSU
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
0001-Pipelining-batch-support-for-libpq-v3.patch (application/octet-stream, 83.7K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index ea7e7da..ac5d4fa 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4637,6 +4637,484 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It somewhat increased client application
+ complexity and extra caution is required to prevent client/server network
+ deadlocks, but can offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ ("ping time") is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 9.6, but clients using it can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-pqbeginbatchmode"><function>PQbeginBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-pqisinbatchmode"><function>PQisInBatchMode(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not allowed. (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQsendEndBatch</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQgetNextQuery</function> to get results. It may eventually exit
+ batch mode with <function>PQendBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it's already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions like
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ etc. The asynchronous requests are followed by a <link
+ linkend="libpq-pqsendendbatch"><function>PQsendEndBatch(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server usually begins executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-pqgetnextquery"><function>PQgetNextQuery</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQgetNextQuery</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQsendEndBatch</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQsendEndBatch</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetNextQuery(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-pqendbatchmode"><function>PQendBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-pqisinbatchmode">
+ <term>
+ <function>PQisInBatchMode</function>
+ <indexterm>
+ <primary>PQisInBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns 1 if a <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>, otherwise 0.
+
+<synopsis>
+int PQisInBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqbeginbatchmode">
+ <term>
+ <function>PQbeginBatchMode</function>
+ <indexterm>
+ <primary>PQbeginBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode and returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+<synopsis>
+int PQbeginBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqendbatchmode">
+ <term>
+ <function>PQendBatchMode</function>
+ <indexterm>
+ <primary>PQendBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results and returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQgetNextQuery</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+<synopsis>
+int PQendBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqsendendbatch">
+ <term>
+ <function>PQsendEndBatch</function>
+ <indexterm>
+ <primary>PQsendEndBatch</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQsendEndBatch(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqgetnextquery">
+ <term>
+ <function>PQgetNextQuery</function>
+ <indexterm>
+ <primary>PQgetNextQuery</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results. Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. See <link
+ linkend="libpq-batch-results">processing results</link>.
+
+<synopsis>
+int PQgetNextQuery(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqqueriesinbatch">
+ <term>
+ <function>PQqueriesInBatch</function>
+ <indexterm>
+ <primary>PQqueriesInBatch</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processsed.
+ This is the number of times <function>PQgetNextQuery</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQqueriesInBatch(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pqbatchisaborted">
+ <term>
+ <function>PQbatchIsAborted</function>
+ <indexterm>
+ <primary>PQbatchIsAborted</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns 1 if the batch curently being received on a
+ <application>libpq</application> connection in <link
+ linkend="libpq-batch-mode">batch mode</link> is
+ <link linkend="libpq-batch-errors">aborted</link>, 0
+ otherwise. The aborted flag is cleared as soon as the result of the
+ <function>PQsendEndBatch</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function as they can tell
+ that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+
+<synopsis>
+int PQbatchIsAborted(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/.gitignore b/src/interfaces/libpq/.gitignore
index cb96af7..4c0d934 100644
--- a/src/interfaces/libpq/.gitignore
+++ b/src/interfaces/libpq/.gitignore
@@ -22,3 +22,4 @@
/encnames.c
/wchar.c
/libpq.rc
+/tmp_check
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 4b1e552..8d5cf21 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -129,6 +129,11 @@ install: all installdirs install-lib
installcheck:
$(MAKE) -C test $@
+check: prove-check
+
+prove-check:
+ $(prove_check)
+
installdirs: installdirs-lib
$(MKDIR_P) '$(DESTDIR)$(includedir)' '$(DESTDIR)$(includedir_internal)' '$(DESTDIR)$(datadir)'
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..1b0b8c5 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,10 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQisInBatchMode 172
+PQqueriesInBatch 173
+PQbeginBatchMode 174
+PQendBatchMode 175
+PQsendEndBatch 176
+PQgetNextQuery 177
+PQbatchIsAborted 178
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index b83af64..71418dd 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3396,6 +3396,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3428,6 +3429,22 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+ queue = conn->cmd_queue_recycle;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+ queue = queue->next;
+ free(prev);
+ }
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index b551875..df48767 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry* PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->in_batch)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,9 +1241,29 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
/* check the arguments */
if (!stmtName)
{
@@ -1287,18 +1319,21 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1343,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1358,6 +1397,81 @@ PQsendQueryPrepared(PGconn *conn,
resultFormat);
}
+/* Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry*
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry*) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/* Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/* Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/* Set up for processing a new query's results */
+static void
+PQstartProcessingNewQuery(PGconn *conn)
+{
+ /* initialize async result-accumulation state */
+ conn->result = NULL;
+ conn->next_result = NULL;
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+}
+
/*
* Common startup code for PQsendQuery and sibling routines
*/
@@ -1377,19 +1491,52 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && !conn->in_batch)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
-
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ if (conn->in_batch)
+ {
+ /* When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQgetNextQuery(...) advances
+ * to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately */
+ PQstartProcessingNewQuery(conn);
+ }
/* ready to send command message */
return true;
@@ -1414,6 +1561,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1574,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1703,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1731,15 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
+
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1866,282 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/* PQisInBatchMode
+ * Return true if currently in batch mode
+ */
+int
+PQisInBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->in_batch;
+}
+
+/* PQqueriesInBatch
+ * Return true if there are queries currently pending in batch mode
+ */
+int
+PQqueriesInBatch(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->cmd_queue_head != NULL;
+}
+
+/* PQbatchIsAborted
+ * Batch being processed is aborted, results discarded until next sync
+ */
+int
+PQbatchIsAborted(PGconn *conn)
+{
+ if (!PQisInBatchMode(conn))
+ return false;
+
+ return conn->batch_aborted;
+}
+
+/* Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * COPY is not permitted in batch mode.
+ *
+ * A set of commands is terminated by a PQsendEndBatch. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQendBatchMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbeginBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->in_batch)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->in_batch = true;
+ conn->batch_aborted = false;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/* End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQendBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (!conn->in_batch)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->in_batch = false;
+ conn->batch_aborted = false;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/* End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQsendEndBatch command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQsendEndBatch(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (!conn->in_batch)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/* PQgetNextQuery
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQgetNextQuery(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return FALSE;
+
+ if (!conn->in_batch)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /* In batch mode but nothing left on the queue; caller can submit
+ * more work or PQendBatchMode() now. */
+ return false;
+ }
+
+ /* Pop the next query from the queue and set up the connection state
+ * as if it'd just been dispatched from a non-batched call */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ PQstartProcessingNewQuery(conn);
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_aborted && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+
+ /* Parse any available data */
+ parseInput(conn);
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1697,7 +2149,6 @@ PQisBusy(PGconn *conn)
* query work remains or an error has occurred (e.g. out of
* memory).
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1749,10 +2200,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->in_batch)
+ {
+ /* batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could still
+ * have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2404,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQexec in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2605,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2623,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->in_batch)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2645,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (!conn->in_batch)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && !conn->in_batch)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2670,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->in_batch)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..32e8c79 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..026453b 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->in_batch)
+ {
+ conn->batch_aborted = false;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -305,7 +313,7 @@ pqParseInput3(PGconn *conn)
* parsing until the application accepts the current
* result.
*/
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
return;
}
break;
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->in_batch)
+ conn->batch_aborted = true;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 1b53d0e..ffdc220 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -93,7 +93,9 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort earlier in a batch */
} ExecStatusType;
typedef enum
@@ -423,6 +425,15 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQisInBatchMode(PGconn *conn);
+extern int PQbatchIsAborted(PGconn *conn);
+extern int PQqueriesInBatch(PGconn *conn);
+extern int PQbeginBatchMode(PGconn *conn);
+extern int PQendBatchMode(PGconn *conn);
+extern int PQsendEndBatch(PGconn *conn);
+extern int PQgetNextQuery(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index e9b73a9..d1ccab0 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,13 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch result,
+ More results expected from this query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -316,6 +320,22 @@ typedef struct pg_conn_host
struct addrinfo *addrlist; /* list of possible backend addresses */
} pg_conn_host;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* PGconn stores all the state data associated with a single connection
* to a backend.
@@ -386,6 +406,8 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ bool in_batch; /* connection is in batch (pipelined) mode */
+ bool batch_aborted; /* current batch is aborted, discarding until next Sync */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -393,6 +415,15 @@ struct pg_conn
PGnotify *notifyHead; /* oldest unreported Notify msg */
PGnotify *notifyTail; /* newest unreported Notify msg */
+ /* The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Support for multiple hosts in connection string */
int nconnhost; /* # of possible hosts */
int whichhost; /* host we're currently considering */
@@ -690,6 +721,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
diff --git a/src/interfaces/libpq/t/001_libpq_async.pl b/src/interfaces/libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..d8f153e
--- /dev/null
+++ b/src/interfaces/libpq/t/001_libpq_async.pl
@@ -0,0 +1,15 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+$node->stop('fast');
diff --git a/src/test/examples/.gitignore b/src/test/examples/.gitignore
index 1957ec1..7a72420 100644
--- a/src/test/examples/.gitignore
+++ b/src/test/examples/.gitignore
@@ -4,3 +4,4 @@
/testlibpq4
/testlo
/testlo64
+/testlibpqbatch
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index 31da210..92a6faf 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
override LDLIBS := $(libpq_pgport) $(LDLIBS)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64 testlibpqbatch
all: $(PROGS)
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..7f1b8b0
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check: prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..28c3eb5
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 5;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..a3c0a33
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1386 @@
+/*
+ * src/test/examples/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution funtionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+
+#ifndef VERBOSE
+#define VERBOSE 0
+#endif
+
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQgetNextQuery when there might still be pending results */
+ if (PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQgetNextQuery() call\n");
+ goto fail;
+ }
+
+ if (PQendBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendEndBatch(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (!PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (!PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchIsAborted(conn))
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "PQgetNextQuery() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (!PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQisInBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbeginBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent BEGIN\n");
+#endif
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent DROP\n");
+#endif
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent CREATE\n");
+#endif
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "sent PREPARE\n");
+#endif
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQgetNextQuery(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "next query!\n");
+#endif
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+#if VERBOSE
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+#endif
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+#if VERBOSE
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+#endif
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+#if VERBOSE
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+#endif
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+#if VERBOSE
+ fprintf(stdout, "sent COMMIT\n");
+#endif
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendEndBatch(conn))
+ {
+#if VERBOSE
+ fprintf(stdout, "Dispatched end batch message\n");
+#endif
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQendBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[MAXINTLEN+2];
+ int formatted = snprintf(&buf[0], MAXINTLEN+1, "%d\n", nrows);
+ if (formatted >= MAXINTLEN+1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long)elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long)elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long)elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognised test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2017-02-17 05:17 ` Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
1 sibling, 1 reply; 182+ messages in thread
From: Prabakaran, Vaishnavi @ 2017-02-17 05:17 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; +Cc: Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Michael Paquier <michael.paquier@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 22 November 2016 at 18:32, Craig Ringer<craig@2ndquadrant.com> wrote:
> On 22 November 2016 at 15:14, Haribabu Kommi
> <kommi.haribabu@gmail.com> wrote:
> >
> > On Fri, Nov 18, 2016 at 7:18 PM, Craig Ringer <craig@2ndquadrant.com>
> wrote:
> >>
> >> The latest is what's attached upthread and what's in the git repo
> >> also referenced upthread.
> >>
> >> I haven't been able to update in response to more recent review due
> >> to other development commitments. At this point I doubt I'll be able
> >> to get update it again in time for v10, so anyone who wants to adopt
> >> it is welcome.
> >
> >
> > Currently patch status is marked as "returned with feedback" in the
> > 11-2016 commitfest. Anyone who wants to work on it can submit the
> > updated patch by taking care of all review comments and change the
> > status of the patch at any time.
> >
> > Thanks for the patch.
>
> Thanks. Sorry I haven't had time to work on it. Priorities.
Hi,
I am interested in this patch and addressed various below comments from reviewers. And, I have separated out code and test module into 2 patches. So, If needed, test patch can be enhanced more, meanwhile code patch can be committed.
>Renaming and refactoring new APIs
> +PQisInBatchMode 172
>+PQqueriesInBatch 173
>+PQbeginBatchMode 174
>+PQendBatchMode 175
>+PQsendEndBatch 176
>+PQgetNextQuery 177
>+PQbatchIsAborted 178
>This set of routines is a bit inconsistent. Why not just prefixing them with PQbatch? Like that for example:
> PQbatchStatus(): consists of disabled/inactive/none, active, error. This covers both PQbatchIsAborted() and PQisInBatchMode().
>PQbatchBegin()
>PQbatchEnd()
>PQbatchQueueSync() or PQbatchQueueFlush, same as PQsendEndBatch() to add and process a sync message into the queue.
Renamed and modified batch status APIs as below
PQisInBatchMode & PQbatchIsAborted ==> PQbatchStatus
PQqueriesInBatch ==> PQbatchQueueCount
PQbeginBatchMode ==> PQbatchBegin
PQendBatchMode ==> PQbatchEnd
PQsendEndBatch ==> PQbatchQueueSync
PQgetNextQuery ==> PQbatchQueueProcess
>PQbatchQueueCount(): returns N>0 if there are N entries, 0 if empty,-1 on failure
>PQbatchQueueProcess(): returns 1 if process can begin, 0 if not, -1 on failure (OOM)
I think it is still ok to keep the current behaviour like other ones present in the same file. E.g:"PQsendPrepare" "PQsendQueryGuts"
>PQqueriesInBatch() (Newname(NN):PQbatchQueueCount)doesn't work as documented.
>It says:
>"Returns the number of queries still in the queue for this batch"
>but in fact it's implemented as a Boolean.
Modified the logic to count number of entries in pending queue and return the count
>The changes in src/test/examples/ are not necessary anymore. You moved all the tests to test_libpq (for the best actually).
Removed these unnecessary changes from src/test/examples folder and corrected the path mentioned in comments section of testlibpqbatch.c
> + while (queue != NULL)
>+ {
> PGcommandQueueEntry *prev = queue;
>+ queue = queue->next;
>+ free(prev);
>+ }
>This should free prev->query.
Both prev->query and prev is freed. Also, this applies to "cmd_queue_recycle" too.
>Running directly make check in src/test/modules/test_libpq/ does not work
Modified "check" rule in makefile
>You could just remove the VERBOSE flag in the tests, having a test more talkative is always better.
Removed ifdef VERBOSE checks.
>But with the libpq batch API, maybe this could be modernized
>with meta-commands like this:
>\startbatch
>...
>\endbatch
I think it is a separate patch candidate.
> It is possible to guess each one of those errors(occurred in PQbatchQueueProcess API) with respectively
> PQgetResult == NULL, PQisInBatchMode() and PQqueriesInBatch().
> Definitely it should be mentioned in the docs that it is possible to make a difference between all those states.
Updated documentation section of PQbatchQueueProcess() with these details.
> + entry = PQmakePipelinedCommand(conn);
>+ entry->queryclass = PGQUERY_SYNC;
>+ entry->query = NULL;
>PQmakePipelinedCommand() returns NULL, and boom.
Corrected to return false if PQmakePipelinedCommand() returns NULL.
> + bool in_batch; /* connection is in batch (pipelined) mode */
>+ bool batch_aborted; /* current batch is aborted, discarding until next Sync */
>Having only one flag would be fine. batch_aborted is set of used only
>when in_batch is used, so both have a strong link
Yes, agree that tracking the batch status via one flag is more clean
So, Added new enum typedef enum
{
PQBATCH_MODE_OFF,
PQBATCH_MODE_ON,
PQBATCH_MODE_ABORTED
} PQBatchStatus;
and " PQBatchStatus batch_status" member of pg_conn is used to track the status of batch mode.
>/* OK, it's launched! */
>- conn->asyncStatus = PGASYNC_BUSY;
>+ if (conn->in_batch)
>+ PQappendPipelinedCommand(conn, pipeCmd);
>+ else
>+ conn->asyncStatus = PGASYNC_BUSY;
>No, it is put in the queue
Though it is put in queue, technically the command is sent to server and server might have started executing it.
So it is ok to use launched I think.
> It may be a good idea to add a test for COPY and trigger a failure.
Added new test - "copy_failure" to trigger failures during copy state.
Please note that COPY is allowed in batch mode, but only syncing batch/queuing any command while copy is in progress is blocked due to potential failure of entire batch.
So updated the documentation for more clarity in this case.
>If I read the code correctly, it seems to me that it is possible to enable the batch mode, and then to use PQexec(), PQsendPrepare will
>just happily process queue the command. Shouldn't PQexec() be prevented in batch mode? Similar remark for PQexecParams(),
>PQexecPrepared() PQdescribePrepared and PQprepare(). In short everything calling PQexecFinish().
Check to verify whether the connection is in batch mode or not is already present in PQexecStart().
And, all the functions calling PQexecFinish() will not be allowed in batch mode anyways. So, no new check is needed I think.
> It may be a good idea to check for PG_PROTOCOL_MAJOR < 3 and issue an error for the new routines.
All the APIs which supports asynchronous command processing can be executed only if PG_PROTOCOL_MAJOR >= 3. So, adding it to new routines are not really needed.
> + After entering batch mode the application dispatches requests
>+ using normal asynchronous <application>libpq</application> functions like
>+ <function>PQsendQueryParams</function>,
><function>PQsendPrepare</function>,
>+ etc.
>It may be better to list all the functions here, PQSendQuery
Documentation updated with list of functions - PQsendQueryParams,PQsendPrepare,
PQsendQueryPrepared,PQdescribePortal,PQdescribePrepared,
PQsendDescribePortal,PQsendDescribePrepared.
>1. Typos in document part - diff-typos.txt file contains the typos specified here
>2. git --check is complaining
>3. s/funtionality/functionality/ and s/recognised/recognized/ typos in testlibpqbatch.c file
>4. s/Batching less useful/Batching is less useful in libpq.sgml
>5. + <para>
>+ Much like asynchronous query mode, there is no performance disadvantage to
>+ using batching and pipelining. It somewhat increased client application
>+ complexity and extra caution is required to prevent client/server network
>+ deadlocks, but can offer considerable performance improvements.
>+ </para>
>I would reword that a bit "it increases client application complexity
>and extra caution is required to prevent client/server deadlocks but
>offers considerable performance improvements".
>6. + ("ping time") is high, and when many small operations are being performed in
>Nit: should use <quote> here. Still not quoting it would be fine.
>7. Postgres 10, not 9.6.
Corrected.
Thanks & Regards,
Vaishnavi
Fujitsu Australia
Disclaimer
The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.
Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.
If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe@fast.au.fujitsu.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments: download all (2 files, .tar.gz)
0001-Pipelining-batch-support-for-libpq-code-v3.patch (application/octet-stream, 48.3K) download
---
doc/src/sgml/libpq.sgml | 463 ++++++++++++++++++++++++++++
src/interfaces/libpq/.gitignore | 1 +
src/interfaces/libpq/Makefile | 5 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 23 ++
src/interfaces/libpq/fe-exec.c | 590 +++++++++++++++++++++++++++++++++---
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 17 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
10 files changed, 1133 insertions(+), 43 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..c80d0d1 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,469 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQdescribePortal</function>,
+ <function>PQdescribePrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server usually begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns PQBATCH_MODE_OFF or PQBATCH_MODE_ON based on a <application>libpq</application> connection's <link
+ linkend="libpq-batch-mode">batch mode</link> status.
+ Returns PQBATCH_MODE_ABORTED if a <application>libpq</application> connection is in aborted status.
+ The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode and returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results and returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results. Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/.gitignore b/src/interfaces/libpq/.gitignore
index cb96af7..4c0d934 100644
--- a/src/interfaces/libpq/.gitignore
+++ b/src/interfaces/libpq/.gitignore
@@ -22,3 +22,4 @@
/encnames.c
/wchar.c
/libpq.rc
+/tmp_check
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 4b1e552..8d5cf21 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -129,6 +129,11 @@ install: all installdirs install-lib
installcheck:
$(MAKE) -C test $@
+check: prove-check
+
+prove-check:
+ $(prove_check)
+
installdirs: installdirs-lib
$(MKDIR_P) '$(DESTDIR)$(includedir)' '$(DESTDIR)$(includedir_internal)' '$(DESTDIR)$(datadir)'
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 65b7c31..4d49f6e 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3428,6 +3428,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3460,6 +3461,28 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+ queue = conn->cmd_queue_recycle;
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index b551875..2aef492 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,9 +1241,29 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
/* check the arguments */
if (!stmtName)
{
@@ -1287,18 +1319,21 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1343,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1358,8 +1397,90 @@ PQsendQueryPrepared(PGconn *conn,
resultFormat);
}
-/*
- * Common startup code for PQsendQuery and sibling routines
+/* PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/* PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/* PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/* PQstartProcessingNewQuery
+ * Set up for processing a new query's results
+ */
+static void
+PQstartProcessingNewQuery(PGconn *conn)
+{
+ /* initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+}
+
+/* PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1498,54 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
-
- /* reset single-row processing mode */
- conn->singleRowMode = false;
-
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately */
+ PQstartProcessingNewQuery(conn);
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1569,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1582,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1711,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1739,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1873,288 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/* PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/* PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/* PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/* PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/* PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/* PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ PQstartProcessingNewQuery(conn);
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+
+ /* Parse any available data */
+ parseInput(conn);
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2214,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2419,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQexec in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2620,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2638,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2660,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2685,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..32e8c79 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->in_batch)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..026453b 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->in_batch)
+ {
+ conn->batch_aborted = false;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -305,7 +313,7 @@ pqParseInput3(PGconn *conn)
* parsing until the application accepts the current
* result.
*/
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
return;
}
break;
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->in_batch)
+ conn->batch_aborted = true;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index e9b73a9..5792247 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -299,6 +305,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -386,6 +408,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -398,6 +421,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -690,6 +723,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v3.patch (application/octet-stream, 43.4K) download
---
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1608 ++++++++++++++++++++++
5 files changed, 1665 insertions(+)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..706b2de
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+#$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..9b72960
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1608 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.7.4.windows.1
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
@ 2017-02-21 07:51 ` Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Michael Paquier @ 2017-02-21 07:51 UTC (permalink / raw)
To: Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; +Cc: Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Daniel Verite <daniel@manitou-mail.org>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Fri, Feb 17, 2017 at 2:17 PM, Prabakaran, Vaishnavi
<VaishnaviP@fast.au.fujitsu.com> wrote:
> On 22 November 2016 at 18:32, Craig Ringer<craig@2ndquadrant.com> wrote:
> I am interested in this patch and addressed various below comments from reviewers. And, I have separated out code and test module into 2 patches. So, If needed, test patch can be enhanced more, meanwhile code patch can be committed.
Cool. Nice to see a new version of this patch.
(You may want to your replies breath a bit, for example by adding an
extra newline to the paragraph you are answering to.)
>>Renaming and refactoring new APIs
>> +PQisInBatchMode 172
>>+PQqueriesInBatch 173
>>+PQbeginBatchMode 174
>>+PQendBatchMode 175
>>+PQsendEndBatch 176
>>+PQgetNextQuery 177
>>+PQbatchIsAborted 178
>>This set of routines is a bit inconsistent. Why not just prefixing them with PQbatch? Like that for example:
>> PQbatchStatus(): consists of disabled/inactive/none, active, error. This covers both PQbatchIsAborted() and PQisInBatchMode().
>>PQbatchBegin()
>>PQbatchEnd()
>>PQbatchQueueSync() or PQbatchQueueFlush, same as PQsendEndBatch() to add and process a sync message into the queue.
>
> Renamed and modified batch status APIs as below
> PQisInBatchMode & PQbatchIsAborted ==> PQbatchStatus
> PQqueriesInBatch ==> PQbatchQueueCount
> PQbeginBatchMode ==> PQbatchBegin
> PQendBatchMode ==> PQbatchEnd
> PQsendEndBatch ==> PQbatchQueueSync
> PQgetNextQuery ==> PQbatchQueueProcess
Thanks. This seems a way cleaner interface to me. Others may have a
different opinion so let's see if there are some.
>>PQbatchQueueCount(): returns N>0 if there are N entries, 0 if empty,-1 on failure
>>PQbatchQueueProcess(): returns 1 if process can begin, 0 if not, -1 on failure (OOM)
>
> I think it is still ok to keep the current behaviour like other ones present in the same file. E.g:"PQsendPrepare" "PQsendQueryGuts"
>
>>PQqueriesInBatch() (Newname(NN):PQbatchQueueCount)doesn't work as documented.
>>It says:
>>"Returns the number of queries still in the queue for this batch"
>>but in fact it's implemented as a Boolean.
>
> Modified the logic to count number of entries in pending queue and return the count
>
>>The changes in src/test/examples/ are not necessary anymore. You moved all the tests to test_libpq (for the best actually).
> Removed these unnecessary changes from src/test/examples folder and corrected the path mentioned in comments section of testlibpqbatch.c
>>But with the libpq batch API, maybe this could be modernized
>>with meta-commands like this:
>>\startbatch
>>...
>>\endbatch
>
> I think it is a separate patch candidate.
Definitely agreed on that. Let's not complicate things further more.
>> It may be a good idea to check for PG_PROTOCOL_MAJOR < 3 and issue an error for the new routines.
>
> All the APIs which supports asynchronous command processing can be executed only if PG_PROTOCOL_MAJOR >= 3. So, adding it to new routines are not really needed.
OK. Let's not complicate the patch more than necessary.
After an extra lookup at the patch, here are some comments.
In the docs when listing any of the PQBATCH_MODE_* variables, you
should have a markup <literal> around them. It would be cleaner to
make a list of the potential values that can be returned by
PQbatchStatus() using <variablelist>.
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+ conn->cmd_queue_recycle = NULL
This could be useful as a separate routine.
+/* PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
This comment block is not project-like. Please use an empty line as
first line with only "/*". The same comment applies to a bunch of the
routines introduced by this patch.
Not sure there is much point in having PQstartProcessingNewQuery. It
only does two things in two places, so that's not worth the loss in
readability.
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
This setup should happen further down.
conn->in_batch is undefined, causing the patch to fail to compile. And
actually you should not need it.
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
return;
This should really not be changed, or it should be changed to a status
dedicated to batching only if batch mode is activated.
If you are planning for integrating this patch into Postres 10, please
be sure that this is registered in the last commit fest that will
begin next week:
https://commitfest.postgresql.org/13/
I'll try to book a couple of cycles to look at it if you are able to
register it into the CF and provide a new version.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2017-02-22 06:14 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-02-23 02:37 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
0 siblings, 2 replies; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-02-22 06:14 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Daniel Verite <daniel@manitou-mail.org>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Tue, Feb 21, 2017 at 6:51 PM, Michael Paquier <michael.paquier@gmail.com>
wrote:
> On Fri, Feb 17, 2017 at 2:17 PM, Prabakaran, Vaishnavi
> <VaishnaviP@fast.au.fujitsu.com> wrote:
> > On 22 November 2016 at 18:32, Craig Ringer<craig@2ndquadrant.com> wrote:
> > I am interested in this patch and addressed various below comments from
> reviewers. And, I have separated out code and test module into 2 patches.
> So, If needed, test patch can be enhanced more, meanwhile code patch can be
> committed.
>
> Cool. Nice to see a new version of this patch.
> (You may want to your replies breath a bit, for example by adding an
> extra newline to the paragraph you are answering to.)
>
> >>Renaming and refactoring new APIs
> >> +PQisInBatchMode 172
> >>+PQqueriesInBatch 173
> >>+PQbeginBatchMode 174
> >>+PQendBatchMode 175
> >>+PQsendEndBatch 176
> >>+PQgetNextQuery 177
> >>+PQbatchIsAborted 178
> >>This set of routines is a bit inconsistent. Why not just prefixing them
> with PQbatch? Like that for example:
> >> PQbatchStatus(): consists of disabled/inactive/none, active, error.
> This covers both PQbatchIsAborted() and PQisInBatchMode().
> >>PQbatchBegin()
> >>PQbatchEnd()
> >>PQbatchQueueSync() or PQbatchQueueFlush, same as PQsendEndBatch() to add
> and process a sync message into the queue.
> >
> > Renamed and modified batch status APIs as below
> > PQisInBatchMode & PQbatchIsAborted ==> PQbatchStatus
> > PQqueriesInBatch ==> PQbatchQueueCount
> > PQbeginBatchMode ==> PQbatchBegin
> > PQendBatchMode ==> PQbatchEnd
> > PQsendEndBatch ==> PQbatchQueueSync
> > PQgetNextQuery ==> PQbatchQueueProcess
>
> Thanks. This seems a way cleaner interface to me. Others may have a
> different opinion so let's see if there are some.
>
>>PQbatchQueueCount(): returns N>0 if there are N entries, 0 if empty,-1 on
> failure
> >>PQbatchQueueProcess(): returns 1 if process can begin, 0 if not, -1 on
> failure (OOM)
> >
> > I think it is still ok to keep the current behaviour like other ones
> present in the same file. E.g:"PQsendPrepare" "PQsendQueryGuts"
> >
> >>PQqueriesInBatch() (Newname(NN):PQbatchQueueCount)doesn't work as
> documented.
> >>It says:
> >>"Returns the number of queries still in the queue for this batch"
> >>but in fact it's implemented as a Boolean.
> >
> > Modified the logic to count number of entries in pending queue and
> return the count
> >
> >>The changes in src/test/examples/ are not necessary anymore. You moved
> all the tests to test_libpq (for the best actually).
> > Removed these unnecessary changes from src/test/examples folder and
> corrected the path mentioned in comments section of testlibpqbatch.c
>
> >>But with the libpq batch API, maybe this could be modernized
> >>with meta-commands like this:
> >>\startbatch
> >>...
> >>\endbatch
> >
> > I think it is a separate patch candidate.
>
> Definitely agreed on that. Let's not complicate things further more.
>
> >> It may be a good idea to check for PG_PROTOCOL_MAJOR < 3 and issue an
> error for the new routines.
> >
> > All the APIs which supports asynchronous command processing can be
> executed only if PG_PROTOCOL_MAJOR >= 3. So, adding it to new routines are
> not really needed.
>
> OK. Let's not complicate the patch more than necessary.
>
> After an extra lookup at the patch, here are some comments.
>
Thanks for reviewing the patch.
>
> In the docs when listing any of the PQBATCH_MODE_* variables, you
> should have a markup <literal> around them. It would be cleaner to
> make a list of the potential values that can be returned by
> PQbatchStatus() using <variablelist>.
>
Modified the format of PQbatchStatus() and other batch APIs too in
documentation along with addition of <literal> tags wherever required.
>
> + while (queue != NULL)
> + {
> + PGcommandQueueEntry *prev = queue;
> +
> + queue = queue->next;
> + if (prev->query)
> + free(prev->query);
> + free(prev);
> + }
> + conn->cmd_queue_recycle = NULL
> This could be useful as a separate routine.
>
Ok, Moved this code to new function - PQfreeCommandQueue().
>
> +/* PQmakePipelinedCommand
> + * Get a new command queue entry, allocating it if required. Doesn't add
> it to
> + * the tail of the queue yet, use PQappendPipelinedCommand once the
> command has
> + * been written for that. If a command fails once it's called this, it
> should
> + * use PQrecyclePipelinedCommand to put it on the freelist or release it.
> + *
> This comment block is not project-like. Please use an empty line as
> first line with only "/*". The same comment applies to a bunch of the
> routines introduced by this patch.
>
Corrected this.
>
> Not sure there is much point in having PQstartProcessingNewQuery. It
> only does two things in two places, so that's not worth the loss in
> readability.
>
Yes, removed this function and placed those two things in line.
>
> + if (conn->batch_status != PQBATCH_MODE_OFF)
> + {
> + pipeCmd = PQmakePipelinedCommand(conn);
> +
> + if (pipeCmd == NULL)
> + return 0; /* error msg already set */
> +
> + last_query = &pipeCmd->query;
> + queryclass = &pipeCmd->queryclass;
> + }
> + else
> + {
> + last_query = &conn->last_query;
> + queryclass = &conn->queryclass;
> + }
> This setup should happen further down.
>
Moved it down post other validations in this function.
>
> conn->in_batch is undefined, causing the patch to fail to compile. And
> actually you should not need it.
>
That is an update missed in my previous patch, corrected in the new patch.
>
> - conn->asyncStatus = PGASYNC_READY;
> + conn->asyncStatus = PGASYNC_READY_MORE;
> return;
> This should really not be changed, or it should be changed to a status
> dedicated to batching only if batch mode is activated.
>
Since pqParseInput3() reads all the input data post "Row description"
message, yes, this change is not needed here.
>
> If you are planning for integrating this patch into Postres 10, please
> be sure that this is registered in the last commit fest that will
> begin next week:
> https://commitfest.postgresql.org/13/
Yes, I have created a new patch entry into the commitfest 2017-03 and
attached the latest patch with this e-mail.
>
> I'll try to book a couple of cycles to look at it if you are able to
> register it into the CF and provide a new version.
>
Thanks again.
Regards,
Vaishnavi,
Fujitsu Australia.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments: download all (2 files, .tar.gz)
0001-Pipelining-batch-support-for-libpq-code-v4.patch (application/octet-stream, 49.7K) download
---
doc/src/sgml/libpq.sgml | 511 +++++++++++++++++++++++++++++++
src/interfaces/libpq/.gitignore | 1 +
src/interfaces/libpq/Makefile | 5 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 ++
src/interfaces/libpq/fe-exec.c | 590 ++++++++++++++++++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
10 files changed, 1188 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..899cc5d 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,517 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQdescribePortal</function>,
+ <function>PQdescribePrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/.gitignore b/src/interfaces/libpq/.gitignore
index cb96af7..4c0d934 100644
--- a/src/interfaces/libpq/.gitignore
+++ b/src/interfaces/libpq/.gitignore
@@ -22,3 +22,4 @@
/encnames.c
/wchar.c
/libpq.rc
+/tmp_check
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 4b1e552..8d5cf21 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -129,6 +129,11 @@ install: all installdirs install-lib
installcheck:
$(MAKE) -C test $@
+check: prove-check
+
+prove-check:
+ $(prove_check)
+
installdirs: installdirs-lib
$(MKDIR_P) '$(DESTDIR)$(includedir)' '$(DESTDIR)$(includedir_internal)' '$(DESTDIR)$(datadir)'
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 65b7c31..f6530e7 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3417,6 +3417,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3428,6 +3447,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3460,6 +3480,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index b551875..abae83c 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,300 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+
+ /* Parse any available data */
+ parseInput(conn);
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2220,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2425,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQexec in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2626,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2644,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2666,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2691,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index e9b73a9..5792247 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -299,6 +305,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -386,6 +408,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -398,6 +421,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -690,6 +723,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v3.patch (application/octet-stream, 43.4K) download
---
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1608 ++++++++++++++++++++++
5 files changed, 1665 insertions(+)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..706b2de
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+#$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..9b72960
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1608 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.7.4.windows.1
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-02-23 02:37 ` Craig Ringer <craig@2ndquadrant.com>
1 sibling, 0 replies; 182+ messages in thread
From: Craig Ringer @ 2017-02-23 02:37 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Iwata, Aya <iwata.aya@jp.fujitsu.com>; pgsql-hackers; Andres Freund <andres@anarazel.de>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Daniel Verite <daniel@manitou-mail.org>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Haribabu Kommi <kommi.haribabu@gmail.com>; Dmitry Igrishin <dmitigr@gmail.com>; Manuel Kniep <m.kniep@web.de>; Michael Paquier <michael.paquier@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
On 22 Feb. 2017 14:14, "Vaishnavi Prabakaran" <vaishnaviprabakaran@gmail.com>
wrote:
Thanks for reviewing the patch.
>
Thanks for picking it up! I've wanted to see this process for some time,
but just haven't had the bandwidth for it.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-03-07 16:52 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-08 01:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
1 sibling, 2 replies; 182+ messages in thread
From: Daniel Verite @ 2017-03-07 16:52 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Vaishnavi Prabakaran wrote:
> Yes, I have created a new patch entry into the commitfest 2017-03 and
> attached the latest patch with this e-mail.
Please find attached a companion patch implementing the batch API in
pgbench, exposed as \beginbatch and \endbatch meta-commands
(without documentation).
The idea for now is to make it easier to exercise the API and test
how batching performs. I guess I'll submit the patch separately in
a future CF, depending on when/if the libpq patch goes in.
While developing this, I noted a few things with 0001-v4:
1. lack of initialization for count in PQbatchQueueCount.
Trivial fix:
inline patch 1: src/interfaces/libpq/fe-exec.c (+1/-1 lines) download inline-1.patch
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -1874,7 +1874,7 @@ PQisBusy(PGconn *conn)
int
PQbatchQueueCount(PGconn *conn)
{
- int count;
+ int count = 0;
pgbench-batch-mode-v1.patch (text/plain, 5.4K) download
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6cb5d4..9b2fce8 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -269,7 +269,8 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
+ * CSTATE_WAIT_RESULT state unless in batch mode.
+ * On a \sleep meta-command, the timer is set,
* and we enter the CSTATE_SLEEP state to wait for it to expire. Other
* meta-commands are executed immediately.
*
@@ -1882,11 +1883,24 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- fprintf(stderr, "%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In batch mode, we use asynchronous functions. If a server-side
+ * error occurs, it will be processed later among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2165,7 +2179,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
return;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in batch mode */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -2207,7 +2227,47 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
else
{
- if (pg_strcasecmp(argv[0], "set") == 0)
+ if (pg_strcasecmp(argv[0], "beginbatch") == 0)
+ {
+ /*
+ * In batch mode, we use a workflow based on libpq batch
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "cannot use batch mode with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "already in batch mode");
+ break;
+ }
+ PQbatchBegin(st->con);
+ }
+ else if (pg_strcasecmp(argv[0], "endbatch") == 0)
+ {
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ commandFailed(st, "not in batch mode");
+ break;
+ }
+ if (!PQbatchQueueSync(st->con))
+ {
+ commandFailed(st, "failed to end the batch");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQbatchEnd(st->con) == 0)
+ {
+ /* collect pending results before getting out of batch mode */
+ st->state = CSTATE_WAIT_RESULT;
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "set") == 0)
{
PgBenchExpr *expr = command->expr;
PgBenchValue result;
@@ -2279,6 +2339,7 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
break;
+
/*
* Wait for the current SQL command to complete
*/
@@ -2295,6 +2356,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_ON &&
+ !PQbatchQueueProcess(st->con))
+ {
+ /* no complete result yet in batch mode*/
+ return;
+ }
+
/*
* Read and discard the query result;
*/
@@ -2307,7 +2375,22 @@ doCustom(TState *thread, CState *st, StatsData *agg)
/* OK */
PQclear(res);
discard_response(st);
- st->state = CSTATE_END_COMMAND;
+ /*
+ * In non-batch mode, only one result per command is expected.
+ * In batch mode, keep waiting for results until getting
+ * PGRES_BATCH_END.
+ */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_END_COMMAND;
+ break;
+ case PGRES_BATCH_END:
+ if (PQbatchEnd(st->con) == 1)
+ {
+ /* all results collected, exit out of command and batch mode */
+ st->state = CSTATE_END_COMMAND;
+ }
+ else
+ fprintf(stderr, "client %d to exit batch mode", st->id);
break;
default:
commandFailed(st, PQerrorMessage(st->con));
@@ -3173,6 +3256,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "beginbatch") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "endbatch") == 0 )
+ {
+ if (my_command->argc > 1)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "unexpected argument", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2017-03-08 01:56 ` Craig Ringer <craig@2ndquadrant.com>
1 sibling, 0 replies; 182+ messages in thread
From: Craig Ringer @ 2017-03-08 01:56 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 8 March 2017 at 00:52, Daniel Verite <daniel@manitou-mail.org> wrote:
> Vaishnavi Prabakaran wrote:
>
>> Yes, I have created a new patch entry into the commitfest 2017-03 and
>> attached the latest patch with this e-mail.
>
> Please find attached a companion patch implementing the batch API in
> pgbench, exposed as \beginbatch and \endbatch meta-commands
> (without documentation).
>
> The idea for now is to make it easier to exercise the API and test
> how batching performs. I guess I'll submit the patch separately in
> a future CF, depending on when/if the libpq patch goes in.
That's great, thanks, and thanks also for the fixes. Any chance you
can attach your updated patch?
I looked at modifying psql to support batching when run
non-interactively, but it would've required major restructuring of its
control loop and I ran out of time. I didn't think of modifying
pgbench. Great to see.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2017-03-08 02:04 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-09 21:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
1 sibling, 2 replies; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-03-08 02:04 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Wed, Mar 8, 2017 at 3:52 AM, Daniel Verite <daniel@manitou-mail.org>
wrote:
> Vaishnavi Prabakaran wrote:
>
> > Yes, I have created a new patch entry into the commitfest 2017-03 and
> > attached the latest patch with this e-mail.
>
> Please find attached a companion patch implementing the batch API in
> pgbench, exposed as \beginbatch and \endbatch meta-commands
> (without documentation).
>
> The idea for now is to make it easier to exercise the API and test
> how batching performs. I guess I'll submit the patch separately in
> a future CF, depending on when/if the libpq patch goes in.
>
>
Thanks for the companion patch and here are some comments:
1. I see, below check is used to verify if the connection is not in batch
mode:
if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
But, it is better to use if (PQbatchStatus(st->con) ==
PQBATCH_MODE_OFF) for this verification. Reason is there is one more state
in batch mode - PQBATCH_MODE_ABORTED. So, if the batch mode is in aborted
status, this check will still assume that the connection is not in batch
mode.
In a same way, "if(PQbatchStatus(st->con) == PQBATCH_MODE_ON)" check is
better to be modified as "PQbatchStatus(st->con) != PQBATCH_MODE_OFF".
2. @@ -2207,7 +2227,47 @@ doCustom(TState *thread, CState *st, StatsData
*agg)
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "already in batch mode");
+ break;
+ }
This is not required as below PQbatchBegin() internally does this
verification.
+ PQbatchBegin(st->con);
3. It is better to check the return value of PQbatchBegin() rather than
assuming success. E.g: PQbatchBegin() will return false if the connection
is in copy in/out/both states.
> While developing this, I noted a few things with 0001-v4:
>
> 1. lack of initialization for count in PQbatchQueueCount.
> Trivial fix:
>
> --- a/src/interfaces/libpq/fe-exec.c
> +++ b/src/interfaces/libpq/fe-exec.c
> @@ -1874,7 +1874,7 @@ PQisBusy(PGconn *conn)
> int
> PQbatchQueueCount(PGconn *conn)
> {
> - int count;
> + int count = 0;
> PGcommandQueueEntry *entry;
>
>
Thanks for your review and yes, Corrected.
> 2. misleading error message in PQexecStart. It gets called by a few other
> functions than PQexec, such as PQprepare. As I understand it, the intent
> here is to forbid the synchronous functions in batch mode, so this error
> message should not single out PQexec.
>
> @@ -1932,6 +2425,13 @@ PQexecStart(PGconn *conn)
> if (!conn)
> return false;
>
> + if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status !=
> PQBATCH_MODE_OFF)
> + {
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext("cannot
> PQexec in batch mode\n"));
> + return false;
> + }
> +
>
>
Hmm, this error message goes with the flow of other error messages in the
same function. E.g: "PQexec not allowed during COPY BOTH" . But, anyways I
modified the
error message to be more generic.
> 3. In relation to #2, PQsendQuery() is not forbidden in batch mode
> although I don't think it can work with it, as it's based on the old
> protocol.
>
>
>
It is actually forbidden and you can see the error message "cannot
PQsendQuery in batch mode, use PQsendQueryParams" when you use
PQsendQuery().
Attached the updated patch.
Thanks & Regards,
Vaishnavi
Fujitsu Australia.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments: download all (2 files, .tar.gz)
0001-Pipelining-batch-support-for-libpq-code-v5.patch (application/octet-stream, 49.8K) download
---
doc/src/sgml/libpq.sgml | 511 +++++++++++++++++++++++++++++++
src/interfaces/libpq/.gitignore | 1 +
src/interfaces/libpq/Makefile | 5 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 ++
src/interfaces/libpq/fe-exec.c | 590 ++++++++++++++++++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
10 files changed, 1188 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..899cc5d 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,517 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQdescribePortal</function>,
+ <function>PQdescribePrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/.gitignore b/src/interfaces/libpq/.gitignore
index cb96af7..4c0d934 100644
--- a/src/interfaces/libpq/.gitignore
+++ b/src/interfaces/libpq/.gitignore
@@ -22,3 +22,4 @@
/encnames.c
/wchar.c
/libpq.rc
+/tmp_check
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 4b1e552..8d5cf21 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -129,6 +129,11 @@ install: all installdirs install-lib
installcheck:
$(MAKE) -C test $@
+check: prove-check
+
+prove-check:
+ $(prove_check)
+
installdirs: installdirs-lib
$(MKDIR_P) '$(DESTDIR)$(includedir)' '$(DESTDIR)$(includedir_internal)' '$(DESTDIR)$(datadir)'
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 65b7c31..f6530e7 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3417,6 +3417,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3428,6 +3447,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3460,6 +3480,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index b551875..abae83c 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,300 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+
+ /* Parse any available data */
+ parseInput(conn);
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2220,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2425,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2626,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2644,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2666,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2691,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index e9b73a9..5792247 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -299,6 +305,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -386,6 +408,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -398,6 +421,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -690,6 +723,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v3.patch (application/octet-stream, 43.4K) download
---
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1608 ++++++++++++++++++++++
5 files changed, 1665 insertions(+)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..706b2de
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+#$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..9b72960
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1608 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.7.4.windows.1
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-03-09 21:43 ` Daniel Verite <daniel@manitou-mail.org>
1 sibling, 0 replies; 182+ messages in thread
From: Daniel Verite @ 2017-03-09 21:43 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Vaishnavi Prabakaran wrote:
> if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
>
> But, it is better to use if (PQbatchStatus(st->con) ==
> PQBATCH_MODE_OFF) for this verification. Reason is there is one more state
> in batch mode - PQBATCH_MODE_ABORTED. So, if the batch mode is in aborted
> status, this check will still assume that the connection is not in batch
> mode.
> In a same way, "if(PQbatchStatus(st->con) == PQBATCH_MODE_ON)" check is
> better to be modified as "PQbatchStatus(st->con) != PQBATCH_MODE_OFF".
Agreed, these two tests need to be changed to account for the
PQBATCH_MODE_ABORTED case. Fixed in new patch.
> 2. @@ -2207,7 +2227,47 @@ doCustom(TState *thread, CState *st, StatsData
> *agg)
> + if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
> + {
> + commandFailed(st, "already in batch mode");
> + break;
> + }
>
> This is not required as below PQbatchBegin() internally does this
> verification.
>
> + PQbatchBegin(st->con);
The point of this test is to specifically forbid a sequence like this
\beginbatch
...(no \endbatch)
\beginbatch
...
and according to the doc for PQbatchBegin, it looks like the return
code won't tell us:
"Causes a connection to enter batch mode if it is currently idle or
already in batch mode.
int PQbatchBegin(PGconn *conn);
Returns 1 for success"
My understanding is that "already in batch mode" is not an error.
"Returns 0 and has no effect if the connection is not currently
idle, i.e. it has a result ready, is waiting for more input from
the server, etc. This function does not actually send anything to
the server, it just changes the libpq connection state"
> 3. It is better to check the return value of PQbatchBegin() rather than
> assuming success. E.g: PQbatchBegin() will return false if the connection
> is in copy in/out/both states.
Given point #2 above, I think both tests are needed:
if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
{
commandFailed(st, "already in batch mode");
break;
}
if (PQbatchBegin(st->con) == 0)
{
commandFailed(st, "failed to start a batch");
break;
}
> > 3. In relation to #2, PQsendQuery() is not forbidden in batch mode
> > although I don't think it can work with it, as it's based on the old
> > protocol.
> >
> It is actually forbidden and you can see the error message "cannot
> PQsendQuery in batch mode, use PQsendQueryParams" when you use
> PQsendQuery().
Sorry for blaming the innocent piece of code, looking closer
it's pgbench that does not display the message.
With the simple query protocol, sendCommand() does essentially:
r = PQsendQuery(st->con, sql);
... other stuff...
if (r == 0)
{
if (debug)
fprintf(stderr, "client %d could not send %s\n",
st->id, command->argv[0]);
st->ecnt++;
return false;
}
So only in debug mode does it inform the user that it failed, and
even then, it does not display PQerrorMessage(st->con).
In non-debug mode it's opaque to the user. If it always fail, it appears
to loop forever. The caller has this comment that is relevant to the problem:
if (!sendCommand(st, command))
{
/*
* Failed. Stay in CSTATE_START_COMMAND state, to
* retry. ??? What the point or retrying? Should
* rather abort?
*/
return;
}
I think it doesn't bother anyone up to now because the immediate
failure modes of PQsendQuery() are resource allocation or protocol
failures that tend to never happen.
Anyway currently \beginbatch avoids the problem by checking
whether querymode == QUERY_SIMPLE, to fail gracefully
instead of letting the endless loop happen.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
pgbench-batch-mode-v2.patch (text/plain, 5.5K) download
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6cb5d4..f93673e 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -269,7 +269,8 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
+ * CSTATE_WAIT_RESULT state unless in batch mode.
+ * On a \sleep meta-command, the timer is set,
* and we enter the CSTATE_SLEEP state to wait for it to expire. Other
* meta-commands are executed immediately.
*
@@ -1882,11 +1883,24 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- fprintf(stderr, "%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In batch mode, we use asynchronous functions. If a server-side
+ * error occurs, it will be processed later among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2165,7 +2179,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
return;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in batch mode */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -2207,7 +2227,51 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
else
{
- if (pg_strcasecmp(argv[0], "set") == 0)
+ if (pg_strcasecmp(argv[0], "beginbatch") == 0)
+ {
+ /*
+ * In batch mode, we use a workflow based on libpq batch
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "cannot use batch mode with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "already in batch mode");
+ break;
+ }
+ if (PQbatchBegin(st->con) == 0)
+ {
+ commandFailed(st, "failed to start a batch");
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "endbatch") == 0)
+ {
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ commandFailed(st, "not in batch mode");
+ break;
+ }
+ if (!PQbatchQueueSync(st->con))
+ {
+ commandFailed(st, "failed to end the batch");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQbatchEnd(st->con) == 0)
+ {
+ /* collect pending results before getting out of batch mode */
+ st->state = CSTATE_WAIT_RESULT;
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "set") == 0)
{
PgBenchExpr *expr = command->expr;
PgBenchValue result;
@@ -2279,6 +2343,7 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
break;
+
/*
* Wait for the current SQL command to complete
*/
@@ -2295,6 +2360,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_ON &&
+ !PQbatchQueueProcess(st->con))
+ {
+ /* no complete result yet in batch mode*/
+ return;
+ }
+
/*
* Read and discard the query result;
*/
@@ -2307,7 +2379,22 @@ doCustom(TState *thread, CState *st, StatsData *agg)
/* OK */
PQclear(res);
discard_response(st);
- st->state = CSTATE_END_COMMAND;
+ /*
+ * In non-batch mode, only one result per command is expected.
+ * In batch mode, keep waiting for results until getting
+ * PGRES_BATCH_END.
+ */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_END_COMMAND;
+ break;
+ case PGRES_BATCH_END:
+ if (PQbatchEnd(st->con) == 1)
+ {
+ /* all results collected, exit out of command and batch mode */
+ st->state = CSTATE_END_COMMAND;
+ }
+ else
+ fprintf(stderr, "client %d to exit batch mode", st->id);
break;
default:
commandFailed(st, PQerrorMessage(st->con));
@@ -3173,6 +3260,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "beginbatch") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "endbatch") == 0 )
+ {
+ if (my_command->argc > 1)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "unexpected argument", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-03-10 13:52 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
1 sibling, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2017-03-10 13:52 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Hi,
I notice that PQsetSingleRowMode() doesn't work when getting batch results.
The function is documented as:
" int PQsetSingleRowMode(PGconn *conn);
This function can only be called immediately after PQsendQuery or one
of its sibling functions, before any other operation on the connection
such as PQconsumeInput or PQgetResult"
But PQbatchQueueProcess() unconditionally clears conn->singleRowMode,
so whatever it was when sending the query gets lost, and besides
other queries might have been submitted in the meantime.
Also if trying to set that mode when fetching like this
while (QbatchQueueProcess(conn)) {
r = PQsetSingleRowMode(conn);
if (r!=1) {
fprintf(stderr, "PQsetSingleRowMode() failed");
}
..
it might work the first time, but on the next iterations, conn->asyncStatus
might be PGASYNC_READY, which is a failure condition for
PQsetSingleRowMode(), so that won't do.
ISTM that the simplest fix would be that when in batch mode,
PQsetSingleRowMode() should register that the last submitted query
does request that mode, and when later QbatchQueueProcess dequeues
the batch entry for the corresponding query, this flag would be popped off
and set as the current mode.
Please find attached the suggested fix, against the v5 of the patch.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
fix-for-singlerowmode.diff (text/x-patch, 2.0K) download
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 3c0be46..8ddf63d 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -1425,7 +1425,7 @@ PQmakePipelinedCommand(PGconn *conn)
}
entry->next = NULL;
entry->query = NULL;
-
+ entry->singleRowMode = false;
return entry;
}
@@ -1783,16 +1783,28 @@ PQsetSingleRowMode(PGconn *conn)
/*
* Only allow setting the flag when we have launched a query and not yet
* received any results.
+ * In batch mode, store the flag in the queue for applying it later.
*/
if (!conn)
return 0;
- if (conn->asyncStatus != PGASYNC_BUSY)
- return 0;
if (conn->queryclass != PGQUERY_SIMPLE &&
conn->queryclass != PGQUERY_EXTENDED)
return 0;
if (conn->result)
return 0;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (conn->asyncStatus != PGASYNC_BUSY)
+ return 0;
+ }
+ else
+ {
+ /* apply to the last submitted query in the batch, or fail */
+ if (conn->cmd_queue_tail != NULL)
+ conn->cmd_queue_tail->singleRowMode = true;
+ else
+ return 0;
+ }
/* OK, set flag */
conn->singleRowMode = true;
@@ -2120,9 +2132,8 @@ PQbatchQueueProcess(PGconn *conn)
* Initialize async result-accumulation state */
pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
-
+ /* Set single-row processing mode */
+ conn->singleRowMode = next_query->singleRowMode;
conn->last_query = next_query->query;
next_query->query = NULL;
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 33f212f..af4f753 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -315,6 +315,7 @@ typedef struct pgCommandQueueEntry
{
PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
char *query; /* SQL command, or NULL if unknown */
+ bool singleRowMode; /* apply single row mode to this query */
struct pgCommandQueueEntry *next;
} PGcommandQueueEntry;
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2017-03-13 00:54 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 01:32 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
0 siblings, 2 replies; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-03-13 00:54 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Sat, Mar 11, 2017 at 12:52 AM, Daniel Verite <daniel@manitou-mail.org>
wrote:
> Hi,
>
> I notice that PQsetSingleRowMode() doesn't work when getting batch results.
>
> The function is documented as:
> " int PQsetSingleRowMode(PGconn *conn);
>
> This function can only be called immediately after PQsendQuery or one
> of its sibling functions, before any other operation on the connection
> such as PQconsumeInput or PQgetResult"
>
> But PQbatchQueueProcess() unconditionally clears conn->singleRowMode,
> so whatever it was when sending the query gets lost, and besides
> other queries might have been submitted in the meantime.
>
> Also if trying to set that mode when fetching like this
>
> while (QbatchQueueProcess(conn)) {
> r = PQsetSingleRowMode(conn);
> if (r!=1) {
> fprintf(stderr, "PQsetSingleRowMode() failed");
> }
> ..
>
> it might work the first time, but on the next iterations, conn->asyncStatus
> might be PGASYNC_READY, which is a failure condition for
> PQsetSingleRowMode(), so that won't do.
>
Thanks for investigating the problem, and could you kindly explain what
"next iteration" you mean here? Because I don't see any problem in
following sequence of calls - PQbatchQueueProcess(),PQsetSingleRowMode()
, PQgetResult(). Am I missing anything?
Please note that it is MUST to call PQgetResult immediately after
PQbatchQueueProcess() as documented.
>
> ISTM that the simplest fix would be that when in batch mode,
> PQsetSingleRowMode() should register that the last submitted query
> does request that mode, and when later QbatchQueueProcess dequeues
> the batch entry for the corresponding query, this flag would be popped off
> and set as the current mode.
>
> Please find attached the suggested fix, against the v5 of the patch.
>
Before going with this fix, I would like you to consider the option of
asking batch processing users(via documentation) to set single-row mode
before calling PQgetResult().
Either way we need to fix the documentation part, letting users know how
they can activate single-row mode while using batch processing.
Let me know your thoughts.
Best Regards,
Vaishnavi,
Fujitsu Australia.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-03-13 01:32 ` Craig Ringer <craig@2ndquadrant.com>
1 sibling, 0 replies; 182+ messages in thread
From: Craig Ringer @ 2017-03-13 01:32 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 13 March 2017 at 08:54, Vaishnavi Prabakaran
<vaishnaviprabakaran@gmail.com> wrote:
> Before going with this fix, I would like you to consider the option of
> asking batch processing users(via documentation) to set single-row mode
> before calling PQgetResult().
> Either way we need to fix the documentation part, letting users know how
> they can activate single-row mode while using batch processing.
> Let me know your thoughts.
Thanks for looking into this, it's clear that I didn't cover the combo
of single-row-mode and batch mode adequately.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-03-13 17:19 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
1 sibling, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2017-03-13 17:19 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Vaishnavi Prabakaran wrote:
> > while (QbatchQueueProcess(conn)) {
> > r = PQsetSingleRowMode(conn);
> > if (r!=1) {
> > fprintf(stderr, "PQsetSingleRowMode() failed");
> > }
> > ..
> Thanks for investigating the problem, and could you kindly explain what
> "next iteration" you mean here? Because I don't see any problem in
> following sequence of calls - PQbatchQueueProcess(),PQsetSingleRowMode()
> , PQgetResult()
I mean the next iteration of the above while statement. Referring
to the doc, that would be the "next batch entry":
" To get the result of the first batch entry the client must call
PQbatchQueueProcess. It must then call PQgetResult and handle the
results until PQgetResult returns null (or would return null if
called). The result from the next batch entry may then be retrieved
using PQbatchQueueProcess and the cycle repeated"
Attached is a bare-bones testcase showing the problem.
As it is, it works, retrieving results for three "SELECT 1"
in the same batch. Now in order to use the single-row
fetch mode, consider adding this:
r = PQsetSingleRowMode(conn);
if (r!=1) {
fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
}
When inserted after the call to PQbatchQueueProcess,
which is what I understand you're saying works for you,
it fails for me when starting to get the results of the 2nd query
and after.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
test-singlerow-batch.c (application/octet-stream, 981B) download
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2017-03-14 06:50 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 02:42 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
0 siblings, 2 replies; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-03-14 06:50 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Tue, Mar 14, 2017 at 4:19 AM, Daniel Verite <daniel@manitou-mail.org>
wrote:
>
> I mean the next iteration of the above while statement. Referring
> to the doc, that would be the "next batch entry":
>
> " To get the result of the first batch entry the client must call
> PQbatchQueueProcess. It must then call PQgetResult and handle the
> results until PQgetResult returns null (or would return null if
> called). The result from the next batch entry may then be retrieved
> using PQbatchQueueProcess and the cycle repeated"
>
> Attached is a bare-bones testcase showing the problem.
> As it is, it works, retrieving results for three "SELECT 1"
> in the same batch. Now in order to use the single-row
> fetch mode, consider adding this:
>
> r = PQsetSingleRowMode(conn);
> if (r!=1) {
> fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
> }
>
> When inserted after the call to PQbatchQueueProcess,
> which is what I understand you're saying works for you,
> it fails for me when starting to get the results of the 2nd query
> and after.
>
>
Thanks for explaining the issue in detail and yes, I do see the issue using
your attached test file.
And I think the problem is with the "parseInput" call at the end of
PQbatchQueueProcess().
I don't see the need for "parseInput" to cover the scope of
PQbatchQueueProcess(), also anyways we do it in PQgetResult().
This function call changes the asyncstatus of connection to READY(following
command complete message from backend), so eventually PQsetSingleRowMode()
is failing. So, attached the alternative fix for this issue.
Please share me your thoughts.
I would also like to hear Craig's opinion on it before applying this fix to
the original patch, just to make sure am not missing anything here.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
0002-single-row-mode-fix.patch (application/octet-stream, 702B) download
From bc320c1f03afe2c614dc75e47aa10b39256df0a7 Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Tue, 14 Mar 2017 17:34:32 +1100
Subject: [PATCH 2/2] single row mode fix
---
src/interfaces/libpq/fe-exec.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 3c0be46..129b49a 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -2155,7 +2155,7 @@ PQbatchQueueProcess(PGconn *conn)
conn->asyncStatus = PGASYNC_BUSY;
/* Parse any available data */
- parseInput(conn);
+ /* parseInput(conn); */
}
return true;
--
2.7.4.windows.1
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-03-16 02:42 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
1 sibling, 0 replies; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-03-16 02:42 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Tue, Mar 14, 2017 at 5:50 PM, Vaishnavi Prabakaran <
vaishnaviprabakaran@gmail.com> wrote:
>
>
> On Tue, Mar 14, 2017 at 4:19 AM, Daniel Verite <daniel@manitou-mail.org>
> wrote:
>
>>
>> I mean the next iteration of the above while statement. Referring
>> to the doc, that would be the "next batch entry":
>>
>> " To get the result of the first batch entry the client must call
>> PQbatchQueueProcess. It must then call PQgetResult and handle the
>> results until PQgetResult returns null (or would return null if
>> called). The result from the next batch entry may then be retrieved
>> using PQbatchQueueProcess and the cycle repeated"
>>
>> Attached is a bare-bones testcase showing the problem.
>> As it is, it works, retrieving results for three "SELECT 1"
>> in the same batch. Now in order to use the single-row
>> fetch mode, consider adding this:
>>
>> r = PQsetSingleRowMode(conn);
>> if (r!=1) {
>> fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
>> }
>>
>> When inserted after the call to PQbatchQueueProcess,
>> which is what I understand you're saying works for you,
>> it fails for me when starting to get the results of the 2nd query
>> and after.
>>
>>
>
> Thanks for explaining the issue in detail and yes, I do see the issue
> using your attached test file.
>
> And I think the problem is with the "parseInput" call at the end of
> PQbatchQueueProcess().
> I don't see the need for "parseInput" to cover the scope of
> PQbatchQueueProcess(), also anyways we do it in PQgetResult().
> This function call changes the asyncstatus of connection to
> READY(following command complete message from backend), so eventually
> PQsetSingleRowMode() is failing. So, attached the alternative fix for this
> issue.
>
Please share me your thoughts.
>
> I would also like to hear Craig's opinion on it before applying this fix
> to the original patch, just to make sure am not missing anything here.
>
>
Attached the code patch applied with the fix explained above and moving the
CF status seeking review.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments: download all (2 files, .tar.gz)
0001-Pipelining-batch-support-for-libpq-code-v6.patch (application/octet-stream, 49.5K) download
From 1d6efd751051399555ec8cb902500421e8f7468b Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Thu, 16 Mar 2017 13:32:08 +1100
Subject: [PATCH] Pipelining-batch-support-for-libpq-code-v6
---
doc/src/sgml/libpq.sgml | 519 +++++++++++++++++++++++++++++++
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 ++
src/interfaces/libpq/fe-exec.c | 587 ++++++++++++++++++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
8 files changed, 1187 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..e82721e 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,525 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQdescribePortal</function>,
+ <function>PQdescribePrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchQueueProcess</function>. This mode selection is effective
+ only for the currently executing query. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 27155f8..bb10b89 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3459,6 +3459,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3470,6 +3489,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3502,6 +3522,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9decd53..7c02f00 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,297 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2217,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2422,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2623,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2641,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2663,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2688,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index b8ec341..33f212f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -384,6 +406,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -693,6 +726,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v3.patch (application/octet-stream, 43.4K) download
---
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1608 ++++++++++++++++++++++
5 files changed, 1665 insertions(+)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..706b2de
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+#$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..9b72960
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1608 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.7.4.windows.1
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-03-16 13:37 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
1 sibling, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2017-03-16 13:37 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Vaishnavi Prabakaran wrote:
> So, attached the alternative fix for this issue.
> Please share me your thoughts.
I assume you prefer the alternative fix because it's simpler.
> I would also like to hear Craig's opinion on it before applying this fix
> to the original patch, just to make sure am not missing anything here.
+1
The main question is whether the predicates enforced
by PQsetSingleRowMode() apply in batch mode in all cases
when it's legit to call that function. Two predicates
that may be problematic are:
if (conn->asyncStatus != PGASYNC_BUSY)
return 0;
and
if (conn->result)
return 0;
The general case with batch mode is that, from the doc:
"The client interleaves result processing with sending batch queries"
Note that I've not even tested that here, I've tested
batching a bunch of queries in a first step and getting the results
in a second step.
I am not confident that the above predicates will be true
in all cases. Also your alternative fix assumes that we add
a user-visible exception to PQsetSingleRowMode in batch mode,
whereby it must not be called as currently documented:
"call PQsetSingleRowMode immediately after a successful call of
PQsendQuery (or a sibling function)"
My gut feeling is that it's not the right direction, I prefer making
the single-row a per-query attribute internally and keep
PQsetSingleRowMode's contract unchanged from the
user's perspective.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2017-03-20 01:32 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-24 15:21 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-03-27 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
0 siblings, 3 replies; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-03-20 01:32 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Fri, Mar 17, 2017 at 12:37 AM, Daniel Verite <daniel@manitou-mail.org>
wrote:
> Vaishnavi Prabakaran wrote:
>
> > So, attached the alternative fix for this issue.
> > Please share me your thoughts.
>
> I assume you prefer the alternative fix because it's simpler.
>
I would like add one more reason for this fix, I think "PQsetSingleRowMode"
should be called only when the result is ready to be processed and before
starting to consume result as it is documented currently as follows -
"To enter single-row mode, call PQsetSingleRowMode immediately after a
successful call of PQsendQuery (or a sibling function). This mode selection
is effective only for the currently executing query. Then call PQgetResult
repeatedly..."
I agree that first fix (you shared) will allow user to set single-row mode
after PQsendQuery, but it also allows user to set this mode at any time of
batch processing(not necessarily "immediately after PQsendQuery"), also
"mode selection is effective only for the currently executing query" will
be false. Please note that I don't see any problem with this deviation. I
like to outline that documentation here anyways needs an update/note.
Before going further, I would like to mention that I have modified the
documentation of batch processing( in v6 code patch) as below:
"To enter single-row mode, call PQsetSingleRowMode immediately after a
successful call of PQbatchQueueProcess. This mode selection is effective
only for the currently executing query. For more information on the
use of PQsetSingleRowMode
, refer to Section 33.6, “Retrieving Query Results Row-By-Row”. "
Please let me know if you think this is not enough but wanted to update
section 33.6 also?
>
> > I would also like to hear Craig's opinion on it before applying this fix
> > to the original patch, just to make sure am not missing anything here.
>
> +1
>
> The main question is whether the predicates enforced
> by PQsetSingleRowMode() apply in batch mode in all cases
> when it's legit to call that function. Two predicates
> that may be problematic are:
> if (conn->asyncStatus != PGASYNC_BUSY)
> return 0;
> and
> if (conn->result)
> return 0;
>
> The general case with batch mode is that, from the doc:
> "The client interleaves result processing with sending batch queries"
>
While sending batch queries in middle of result processing, only the query
is appended to the list of queries maintained for batch processing and no
current connection attribute impacting result processing will be changed.
So, calling the PQsetSingleRowMode in-between result processing will fail
as it tries to set single-row mode for currently executing query for which
result processing is already started.
Note that I've not even tested that here,
I've tested
> batching a bunch of queries in a first step and getting the results
> in a second step.
> I am not confident that the above predicates will be true
> in all cases.
Also your alternative fix assumes that we add
> a user-visible exception to PQsetSingleRowMode in batch mode,
> whereby it must not be called as currently documented:
> "call PQsetSingleRowMode immediately after a successful call of
> PQsendQuery (or a sibling function)"
My gut feeling is that it's not the right direction, I prefer making
> the single-row a per-query attribute internally and keep
> PQsetSingleRowMode's contract unchanged from the
> user's perspective.
>
>
Am going to include the test which you shared in the test patch. Please let
me know if you want to cover anymore specific cases to gain confidence.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-03-24 15:21 ` David Steele <david@pgmasters.net>
2017-03-25 12:50 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2 siblings, 1 reply; 182+ messages in thread
From: David Steele @ 2017-03-24 15:21 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Craig Ringer <craig@2ndquadrant.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Hi Vaishnavi,
On 3/19/17 9:32 PM, Vaishnavi Prabakaran wrote:
> On Fri, Mar 17, 2017 at 12:37 AM, Daniel Verite <daniel@manitou-mail.org
>
> Please let me know if you think this is not enough but wanted to update
> section 33.6 also?
Daniel, any input here?
> > I would also like to hear Craig's opinion on it before applying this fix
> > to the original patch, just to make sure am not missing anything here.
Craig?
>
> Am going to include the test which you shared in the test patch. Please
> let me know if you want to cover anymore specific cases to gain
> confidence.
I have marked this submission "Waiting for Author" since it appears a
new patch is required based on input and adding a new test.
--
-David
david@pgmasters.net
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-24 15:21 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
@ 2017-03-25 12:50 ` Craig Ringer <craig@2ndquadrant.com>
2017-03-27 07:26 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2017-03-25 12:50 UTC (permalink / raw)
To: David Steele <david@pgmasters.net>; +Cc: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 24 March 2017 at 23:21, David Steele <david@pgmasters.net> wrote:
> Hi Vaishnavi,
>
> On 3/19/17 9:32 PM, Vaishnavi Prabakaran wrote:
>>
>> On Fri, Mar 17, 2017 at 12:37 AM, Daniel Verite <daniel@manitou-mail.org
>>
>> Please let me know if you think this is not enough but wanted to update
>> section 33.6 also?
>
>
> Daniel, any input here?
>
>> > I would also like to hear Craig's opinion on it before applying this
>> fix
>> > to the original patch, just to make sure am not missing anything
>> here.
>
>
> Craig?
I'm fairly confident that I overlooked single row mode entirely in the
original patch, though it's long enough ago that it's hard for me to
remember exactly.
I don't really have much of an opinion on the best handling of it.
I would expect to be setting single-row mode just before I requested
the *result* from the next pending query, since it relates to result
processing rather than query dispatch. But that's about all the
opinion I have here.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-24 15:21 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-03-25 12:50 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2017-03-27 07:26 ` Michael Paquier <michael.paquier@gmail.com>
2017-03-27 07:42 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Michael Paquier @ 2017-03-27 07:26 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: David Steele <david@pgmasters.net>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Sat, Mar 25, 2017 at 9:50 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
> I'm fairly confident that I overlooked single row mode entirely in the
> original patch, though it's long enough ago that it's hard for me to
> remember exactly.
>
> I don't really have much of an opinion on the best handling of it.
>
> I would expect to be setting single-row mode just before I requested
> the *result* from the next pending query, since it relates to result
> processing rather than query dispatch. But that's about all the
> opinion I have here.
Yeah, I think that it makes sense to allow users to switch to single
row mode before requesting a result in the queue. It seems to me that
this should also be effective only during the fetching of one single
result set. When the client moves on to the next item in the queue we
should make necessary again a call to PQsetSingleRowMode().
Regarding the patch, I have spotted the following things in the last version:
- src/test/modules/Makefile should include test_libpq.
- Regression tests from 0002 are failing:
ok 1 - testlibpqbatch disallowed_in_batch
ok 2 - testlibpqbatch simple_batch
ok 3 - testlibpqbatch multi_batch
ok 4 - testlibpqbatch batch_abort
ok 5 - testlibpqbatch timings
not ok 6 - testlibpqbatch copyfailure
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-24 15:21 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-03-25 12:50 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-03-27 07:26 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2017-03-27 07:42 ` Craig Ringer <craig@2ndquadrant.com>
2017-03-27 08:18 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2017-03-27 07:42 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: David Steele <david@pgmasters.net>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 27 March 2017 at 15:26, Michael Paquier <michael.paquier@gmail.com> wrote:
> On Sat, Mar 25, 2017 at 9:50 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
>> I'm fairly confident that I overlooked single row mode entirely in the
>> original patch, though it's long enough ago that it's hard for me to
>> remember exactly.
>>
>> I don't really have much of an opinion on the best handling of it.
>>
>> I would expect to be setting single-row mode just before I requested
>> the *result* from the next pending query, since it relates to result
>> processing rather than query dispatch. But that's about all the
>> opinion I have here.
>
> Yeah, I think that it makes sense to allow users to switch to single
> row mode before requesting a result in the queue. It seems to me that
> this should also be effective only during the fetching of one single
> result set. When the client moves on to the next item in the queue we
> should make necessary again a call to PQsetSingleRowMode().
>
> Regarding the patch, I have spotted the following things in the last version:
> - src/test/modules/Makefile should include test_libpq.
> - Regression tests from 0002 are failing:
> ok 1 - testlibpqbatch disallowed_in_batch
> ok 2 - testlibpqbatch simple_batch
> ok 3 - testlibpqbatch multi_batch
> ok 4 - testlibpqbatch batch_abort
> ok 5 - testlibpqbatch timings
> not ok 6 - testlibpqbatch copyfailure
There are only a few more days left of this commit fest.
Things are sounding pretty ready here, with some final cleanups
pending. It'd be cool to get this into Pg 10 :)
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-24 15:21 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-03-25 12:50 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-03-27 07:26 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-27 07:42 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2017-03-27 08:18 ` Michael Paquier <michael.paquier@gmail.com>
0 siblings, 0 replies; 182+ messages in thread
From: Michael Paquier @ 2017-03-27 08:18 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: David Steele <david@pgmasters.net>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Mon, Mar 27, 2017 at 4:42 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
> On 27 March 2017 at 15:26, Michael Paquier <michael.paquier@gmail.com> wrote:
>> On Sat, Mar 25, 2017 at 9:50 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
>>> I'm fairly confident that I overlooked single row mode entirely in the
>>> original patch, though it's long enough ago that it's hard for me to
>>> remember exactly.
>>>
>>> I don't really have much of an opinion on the best handling of it.
>>>
>>> I would expect to be setting single-row mode just before I requested
>>> the *result* from the next pending query, since it relates to result
>>> processing rather than query dispatch. But that's about all the
>>> opinion I have here.
>>
>> Yeah, I think that it makes sense to allow users to switch to single
>> row mode before requesting a result in the queue. It seems to me that
>> this should also be effective only during the fetching of one single
>> result set. When the client moves on to the next item in the queue we
>> should make necessary again a call to PQsetSingleRowMode().
>>
>> Regarding the patch, I have spotted the following things in the last version:
>> - src/test/modules/Makefile should include test_libpq.
>> - Regression tests from 0002 are failing:
>> ok 1 - testlibpqbatch disallowed_in_batch
>> ok 2 - testlibpqbatch simple_batch
>> ok 3 - testlibpqbatch multi_batch
>> ok 4 - testlibpqbatch batch_abort
>> ok 5 - testlibpqbatch timings
>> not ok 6 - testlibpqbatch copyfailure
>
> There are only a few more days left of this commit fest.
>
> Things are sounding pretty ready here, with some final cleanups
> pending. It'd be cool to get this into Pg 10 :)
Yes, that's one of the items I'd like to help with by the end of the CF.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-03-27 13:28 ` Daniel Verite <daniel@manitou-mail.org>
2 siblings, 0 replies; 182+ messages in thread
From: Daniel Verite @ 2017-03-27 13:28 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Vaishnavi Prabakaran wrote:
> Please let me know if you think this is not enough but wanted to update
> section 33.6 also?
Yes, if the right place to call PQsetSingleRowMode() is immediately
after PQbatchQueueProcess(), I think we need to update
"33.6. Retrieving Query Results Row-By-Row"
with that information, otherwise what it says is just wrong
when in batch mode.
Also, in 33.5, I suggest to not use the "currently executing
query" as a synonym for the "query currently being processed"
to avoid any confusion between what the backend is executing
and a prior query whose results are being collected by the client
at the same moment.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-03-27 15:27 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2 siblings, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2017-03-27 15:27 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Vaishnavi Prabakaran wrote:
> Am going to include the test which you shared in the test patch. Please let
> me know if you want to cover anymore specific cases to gain confidence.
One bit of functionality that does not work in batch mode and is left
as is by the patch is the PQfn() fast path interface and the large object
functions that use it.
Currently, calling lo_* functions inside a batch will fail with a message
that depends on whether the internal lo_initialize() has been successfully
called before.
If it hasn't, PQerrorMessage() will be:
"Synchronous command execution functions are not allowed in batch mode"
which is fine, but it comes by happenstance from lo_initialize()
calling PQexec() to fetch the function OIDs from pg_catalog.pg_proc.
OTOH, if lo_initialize() has succeeded before, a call to a large
object function will fail with a different message:
"connection in wrong state"
which is emitted by PQfn() based on conn->asyncStatus != PGASYNC_IDLE
Having an unified error message would be more user friendly.
Concerning the doc, when saying in 33.5.2:
"In batch mode only asynchronous operations are permitted"
the server-side functions are indeed ruled out, since PQfn() is
synchronous, but maybe we should be a bit more explicit
about that?
Chapter 34.3 (Large Objects / Client Interfaces) could also
mention the incompatibility with batch mode.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2017-03-28 04:57 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-03-28 04:57 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Thanks Craig and Michael for confirming that "PQsetSingleRowMode" should be
called right after "PQbatchQueueProcess".
Michael Paquier wrote:
> It seems to me that
>this should also be effective only during the fetching of one single
>result set. When the client moves on to the next item in the queue we
>should make necessary again a call to PQsetSingleRowMode().
Yes, the current behavior(with V6 code patch) is exactly the same as you
described above. PQsetSingleRowMode() should be called each time after
"PQbatchQueueProcess"
to set result processing to single-row mode.
>src/test/modules/Makefile should include test_libpq.
Yes, added test_libpq to the list in Makefile.
Daniel Verite wrote:
>> Please let me know if you think this is not enough but wanted to update
>> section 33.6 also?
>Yes, if the right place to call PQsetSingleRowMode() is immediately
>after PQbatchQueueProcess(), I think we need to update
>"33.6. Retrieving Query Results Row-By-Row"
>with that information, otherwise what it says is just wrong
>when in batch mode.
Yes, I have updated Chapter 33.6 by adding note for batch mode.
>Also, in 33.5, I suggest to not use the "currently executing
>query" as a synonym for the "query currently being processed"
>to avoid any confusion between what the backend is executing
>and a prior query whose results are being collected by the client
>at the same moment.
Yes correct, I modified the words to "query currently being processed" as
suggested.
> One bit of functionality that does not work in batch mode and is left
> as is by the patch is the PQfn() fast path interface and the large object
> functions that use it.
>
> Currently, calling lo_* functions inside a batch will fail with a message
> that depends on whether the internal lo_initialize() has been successfully
> called before.
>
> If it hasn't, PQerrorMessage() will be:
> "Synchronous command execution functions are not allowed in batch mode"
> which is fine, but it comes by happenstance from lo_initialize()
> calling PQexec() to fetch the function OIDs from pg_catalog.pg_proc.
>
> OTOH, if lo_initialize() has succeeded before, a call to a large
> object function will fail with a different message:
> "connection in wrong state"
> which is emitted by PQfn() based on conn->asyncStatus != PGASYNC_IDLE
>
> Having an unified error message would be more user friendly.
>
>
Thanks for finding out this and yes, added a new check in PQfn() to give
the same error message - "Synchronous command execution functions are not
allowed in batch mode" when called in batch mode.
> Concerning the doc, when saying in 33.5.2:
> "In batch mode only asynchronous operations are permitted"
> the server-side functions are indeed ruled out, since PQfn() is
> synchronous, but maybe we should be a bit more explicit
> about that?
> Chapter 34.3 (Large Objects / Client Interfaces) could also
> mention the incompatibility with batch mode.
>
>
Updated 33.5.2 to be more clear about what functions are allowed and what
are not allowed. Updated Chapter 33.3(Large Objects/ Client Interfaces) to
let the user know about the incompatibility with batch mode .
Attached the latest patch and here is the RT run result:
ok 1 - testlibpqbatch disallowed_in_batch
ok 2 - testlibpqbatch simple_batch
ok 3 - testlibpqbatch multi_batch
ok 4 - testlibpqbatch batch_abort
ok 5 - testlibpqbatch timings
ok 6 - testlibpqbatch copyfailure
ok 7 - testlibpqbatch test_singlerowmode
ok
All tests successful.
Files=1, Tests=7, 5 wallclock secs ( 0.01 usr 0.00 sys + 1.79 cusr 0.35
csys = 2.15 CPU)
Result: PASS
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments: download all (2 files, .tar.gz)
0001-Pipelining-batch-support-for-libpq-code-v7.patch (application/octet-stream, 50.9K) download
---
doc/src/sgml/libpq.sgml | 529 ++++++++++++++++++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 ++
src/interfaces/libpq/fe-exec.c | 594 ++++++++++++++++++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
9 files changed, 1208 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..3b7e37b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,527 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchQueueProcess</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4695,6 +5216,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchQueueProcess</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 27155f8..bb10b89 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3459,6 +3459,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3470,6 +3489,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3502,6 +3522,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9decd53..b0f5ce7 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,297 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2217,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2422,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2623,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2641,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2663,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2688,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2569,6 +3090,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index b8ec341..33f212f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -384,6 +406,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -693,6 +726,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v4.patch (application/octet-stream, 45.1K) download
---
src/test/modules/Makefile | 1 +
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1661 ++++++++++++++++++++++
6 files changed, 1719 insertions(+)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 3ce9904..310d3ba 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -11,6 +11,7 @@ SUBDIRS = \
snapshot_too_old \
test_ddl_deparse \
test_extensions \
+ test_libpq \
test_parser \
test_pg_dump \
test_rls_hooks \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..add5d23
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure test_singlerowmode);
+#my @tests = qw(test_singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+#$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..1134bd1
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1661 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQbatchBegin(conn);
+
+ for (i=0; i < 3; i++) {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchQueueSync(conn);
+
+ i=0;
+ while (PQbatchQueueProcess(conn))
+ {
+ r = PQsetSingleRowMode(conn);
+ if (r!=1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ else if (est == PGRES_SINGLE_TUPLE)
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ else if (est == PGRES_BATCH_END)
+ fprintf(stderr, ", end of batch reached\n");
+ else if (est != PGRES_COMMAND_OK)
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ PQclear(res);
+ }
+ i++;
+ }
+ PQbatchEnd(conn);
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else if (strcmp(argv[3], "test_singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+
+ if(run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.7.4.windows.1
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-03-28 08:10 ` Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
0 siblings, 1 reply; 182+ messages in thread
From: Michael Paquier @ 2017-03-28 08:10 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Tue, Mar 28, 2017 at 1:57 PM, Vaishnavi Prabakaran
<vaishnaviprabakaran@gmail.com> wrote:
> Thanks Craig and Michael for confirming that "PQsetSingleRowMode" should be
> called right after "PQbatchQueueProcess".
>
> Michael Paquier wrote:
>
>> It seems to me that
>>this should also be effective only during the fetching of one single
>>result set. When the client moves on to the next item in the queue we
>>should make necessary again a call to PQsetSingleRowMode().
>
> Yes, the current behavior(with V6 code patch) is exactly the same as you
> described above. PQsetSingleRowMode() should be called each time after
> "PQbatchQueueProcess" to set result processing to single-row mode.
Okay, that's fine for me then.
> Attached the latest patch and here is the RT run result:
> ok 1 - testlibpqbatch disallowed_in_batch
> ok 2 - testlibpqbatch simple_batch
> ok 3 - testlibpqbatch multi_batch
> ok 4 - testlibpqbatch batch_abort
> ok 5 - testlibpqbatch timings
> ok 6 - testlibpqbatch copyfailure
> ok 7 - testlibpqbatch test_singlerowmode
> ok
> All tests successful.
> Files=1, Tests=7, 5 wallclock secs ( 0.01 usr 0.00 sys + 1.79 cusr 0.35
> csys = 2.15 CPU)
> Result: PASS
ok 1 - testlibpqbatch disallowed_in_batch
ok 2 - testlibpqbatch simple_batch
ok 3 - testlibpqbatch multi_batch
ok 4 - testlibpqbatch batch_abort
ok 5 - testlibpqbatch timings
not ok 6 - testlibpqbatch copyfailure
Hm... Not here. I saw something like that a couple of days ago on my
macos laptop and that was related to a variable not initialized. From
001_libpq_async_main.log:
7-03-28 17:05:49.159 JST [31553] t/001_libpq_async.pl LOG: execute
<unnamed>: copy batch_demo(id) to stdout;
2017-03-28 17:05:49.159 JST [31553] t/001_libpq_async.pl LOG: execute
<unnamed>: copy batch_demo(itemno) FROM stdin;
2017-03-28 17:05:49.160 JST [31553] t/001_libpq_async.pl ERROR:
unexpected message type 0x50 during COPY from stdin
2017-03-28 17:05:49.160 JST [31553] t/001_libpq_async.pl CONTEXT:
COPY batch_demo, line 1
From regress_log_001_libpq_async :
ok 5 - testlibpqbatch timings
# Running: testlibpqbatch dbname=postgres 10000 copyfailure
dispatching SELECT query failed: cannot queue commands during COPY
COPYBUF: 5
Error status and message got from server due to COPY command failure
are : PGRES_FATAL_ERROR ERROR: unexpected message type 0x50 during
COPY from stdin
CONTEXT: COPY batch_demo, line 1
So it seems to me that you are still missing something..
src/test/modules/Makefile | 1 +
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1661 ++++++++++++++++++++++
6 files changed, 1719 insertions(+)
Could you as well update src/tools/msvc/vcregress.pl, aka the routine
modulescheck so as this new test is skipped. I am sure that nobody
will scream if this test is not run on Windows, but the buildfarm will
complain if the patch is let in its current state.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2017-03-28 13:28 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2017-03-28 13:28 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Michael Paquier wrote:
> # Running: testlibpqbatch dbname=postgres 10000 copyfailure
> dispatching SELECT query failed: cannot queue commands during COPY
>
> COPYBUF: 5
>
> Error status and message got from server due to COPY command failure
> are : PGRES_FATAL_ERROR ERROR: unexpected message type 0x50 during
> COPY from stdin
> CONTEXT: COPY batch_demo, line 1
Same result here.
BTW the doc says:
"In batch mode only asynchronous operations are permitted, and COPY is
not recommended as it most likely will trigger failure in batch
processing"
Yet it seems that the test assumes that it should work nonetheless.
I don't quite understand what we expect of this test, given what's
documented. Or what am I missing?
While looking at the regress log, I noticed multiple spurious
test_singlerowmode tests among the others. Should be fixed with:
inline patch 1: src/test/modules/test_libpq/testlibpqbatch.c (+0/-0 lines) download inline-1.patch
--- a/src/test/modules/test_libpq/testlibpqbatch.c
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -1578,6 +1578,7 @@ main(int argc, char **argv)
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2017-03-29 03:40 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-03-29 03:40 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Michael Paquier wrote:
>Could you as well update src/tools/msvc/vcregress.pl, aka the routine
>modulescheck so as this new test is skipped. I am sure that nobody
>will scream if this test is not run on Windows, but the buildfarm will
>complain if the patch is let in its current state.
Thanks for the finding. Hmm, modulescheck will not pick up test_libpq as
"subdircheck" will fail for this module(because it does not have
"sql"/"expected" folders in it) and hence it will be skipped.
But, Modified "Mkvcbuild.pm" to exclude "test_libpq" module, as this test
anyways will not be run in Windows and also because it uses linux specific
APIs(gettimeofday , timersub) .
On Wed, Mar 29, 2017 at 12:28 AM, Daniel Verite <daniel@manitou-mail.org>
wrote:
> Michael Paquier wrote:
>
> > # Running: testlibpqbatch dbname=postgres 10000 copyfailure
> > dispatching SELECT query failed: cannot queue commands during COPY
> >
> > COPYBUF: 5
> >
> > Error status and message got from server due to COPY command failure
> > are : PGRES_FATAL_ERROR ERROR: unexpected message type 0x50 during
> > COPY from stdin
> > CONTEXT: COPY batch_demo, line 1
>
> Same result here.
>
> BTW the doc says:
> "In batch mode only asynchronous operations are permitted, and COPY is
> not recommended as it most likely will trigger failure in batch
> processing"
> Yet it seems that the test assumes that it should work nonetheless.
> I don't quite understand what we expect of this test, given what's
> documented. Or what am I missing?
>
copyfailure test is basically tests the error scenarios arising in batch
mode on using copy command. And, test expects error(not that it should
work) and the test is made to pass if expected error message is received.
So, it is error case testing behaves like:
expects error -> receives error = test pass
expects error -> no error = test fail
There are 2 cases tested here:
1. Example for the case - Using COPY command in batch mode will trigger
failure. (Specified in documentation)
Failure can be seen only when processing the copy command's results. That
is, after dispatching COPY command, server goes into COPY state , but the
client dispatched another query in batch mode.
Below error messages belongs to this case :
Error status and message got from server due to COPY command failure are :
PGRES_FATAL_ERROR ERROR: unexpected message type 0x50 during COPY from
stdin
CONTEXT: COPY batch_demo, line 1
message type 0x5a arrived from server while idle
2. When connection is is copy state, that is, while processing copy
command's result, Cannot queue any query in batch mode.
Below error message belongs to this case:
dispatching SELECT query failed: cannot queue commands during COPY
I added this test following Michael's review comment - "It may be a good
idea to add a test for COPY and trigger a failure."
Hope this clarifies the presence of error message in log file and why the
test is made to pass.
>
> While looking at the regress log, I noticed multiple spurious
> test_singlerowmode tests among the others. Should be fixed with:
>
> --- a/src/test/modules/test_libpq/testlibpqbatch.c
> +++ b/src/test/modules/test_libpq/testlibpqbatch.c
> @@ -1578,6 +1578,7 @@ main(int argc, char **argv)
> run_batch_abort = 0;
> run_timings = 0;
> run_copyfailure = 0;
> + run_singlerowmode = 0;
> if (strcmp(argv[3], "disallowed_in_batch") == 0)
> run_disallowed_in_batch = 1;
> else if (strcmp(argv[3], "simple_batch") == 0)
>
Thanks for finding it out and yes corrected.
> There's also the fact that this test doesn't care whether it fails
> (compare with all the "goto fail" of the other tests).
>
Modified the test to check for failure condition.
>
> And it happens that PQsetSingleRowMode() fails after the call to
> PQbatchQueueProcess() that instantiates a PGresult
> of status PGRES_BATCH_END to indicate the end of the batch,
>
> To me this shows how this way of setting the single row mode is not ideal.
> In order to avoid the failure, the loop should know in advance
> what kind of result it's going to dequeue before calling PQgetResult(),
> which doesn't feel right.
>
>
Yes, it is one of the requirement that the batch mode application should
know the order of queries it sent and the expected results. (Specified in
"Processing results" section of batch mode documentation)
Attached the latest code and test patches.
Thanks & Regards,
Vaishnavi
Fujitusu Australia.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments: download all (2 files, .tar.gz)
0001-Pipelining-batch-support-for-libpq-code-v8.patch (application/octet-stream, 51.1K) download
From 04c69538495ffa10ce89c198d12df042ce5f7084 Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Wed, 29 Mar 2017 13:38:47 +1100
Subject: [PATCH 1/2] Pipelining-batch-support-for-libpq-code-v8
---
doc/src/sgml/libpq.sgml | 529 ++++++++++++++++++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 ++
src/interfaces/libpq/fe-exec.c | 594 ++++++++++++++++++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
9 files changed, 1208 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..3b7e37b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,527 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchQueueProcess</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4695,6 +5216,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchQueueProcess</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 27155f8..bb10b89 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3459,6 +3459,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3470,6 +3489,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3502,6 +3522,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9decd53..b0f5ce7 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,297 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2217,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2422,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2623,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2641,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2663,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2688,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2569,6 +3090,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index b8ec341..33f212f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -384,6 +406,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -693,6 +726,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v5.patch (application/octet-stream, 46.3K) download
From c9985df208c0cf14b5e99d21aa85f133f10572e1 Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Wed, 29 Mar 2017 13:39:05 +1100
Subject: [PATCH 2/2] Pipelining-batch-support-for-libpq-test-v5
---
src/test/modules/Makefile | 1 +
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 25 +
src/test/modules/test_libpq/testlibpqbatch.c | 1688 ++++++++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 2 +-
7 files changed, 1746 insertions(+), 1 deletion(-)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 3ce9904..310d3ba 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -11,6 +11,7 @@ SUBDIRS = \
snapshot_too_old \
test_ddl_deparse \
test_extensions \
+ test_libpq \
test_parser \
test_pg_dump \
test_rls_hooks \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..bff2c0d
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,25 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..7a1c004
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1688 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|copyfailure|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQbatchBegin(conn);
+
+ for (i=0; i < 3; i++) {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchQueueSync(conn);
+
+ i=0;
+ while (PQbatchQueueProcess(conn))
+ {
+ int isSingleTuple = 0;
+ /* Set single row mode for only first 3 SELECT queries */
+ if(i < 3)
+ r = PQsetSingleRowMode(conn);
+ if (r!=1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ if(!isSingleTuple)
+ {
+ fprintf(stderr, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ goto fail;
+ }
+ isSingleTuple=0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ }
+ i++;
+ }
+ PQbatchEnd(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+
+ if(run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index ba1bf6d..4a2a4dd 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -46,7 +46,7 @@ my @contrib_excludes = (
'ltree_plpython', 'pgcrypto',
'sepgsql', 'brin',
'test_extensions', 'test_pg_dump',
- 'snapshot_too_old');
+ 'snapshot_too_old', 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
--
2.7.4.windows.1
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-03-30 01:08 ` Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Michael Paquier @ 2017-03-30 01:08 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Wed, Mar 29, 2017 at 12:40 PM, Vaishnavi Prabakaran
<vaishnaviprabakaran@gmail.com> wrote:
> Michael Paquier wrote:
>>Could you as well update src/tools/msvc/vcregress.pl, aka the routine
>>modulescheck so as this new test is skipped. I am sure that nobody
>>will scream if this test is not run on Windows, but the buildfarm will
>>complain if the patch is let in its current state.
>
> Thanks for the finding. Hmm, modulescheck will not pick up test_libpq as
> "subdircheck" will fail for this module(because it does not have
> "sql"/"expected" folders in it) and hence it will be skipped.
> But, Modified "Mkvcbuild.pm" to exclude "test_libpq" module, as this test
> anyways will not be run in Windows and also because it uses linux specific
> APIs(gettimeofday , timersub).
src/port/gettimeofday.c extends things on Windows, and we could
consider to just drop the timing portion for simplicity (except
Windows I am not seeing any platform missing timersub). But that's
just a point of detail. Or we could just drop those tests from the
final patch, and re-implement them after having some psql-level
subcommands. That would far better for portability.
> There are 2 cases tested here:
>
> 1. Example for the case - Using COPY command in batch mode will trigger
> failure. (Specified in documentation)
> Failure can be seen only when processing the copy command's results. That
> is, after dispatching COPY command, server goes into COPY state , but the
> client dispatched another query in batch mode.
>
> Below error messages belongs to this case :
> Error status and message got from server due to COPY command failure are :
> PGRES_FATAL_ERROR ERROR: unexpected message type 0x50 during COPY from
> stdin
> CONTEXT: COPY batch_demo, line 1
>
> message type 0x5a arrived from server while idle
Such messages are really confusing to the user as they refer to the
protocol going out of sync. I would think that something like "cannot
process COPY results during a batch processing" would be cleaner.
Isn't some more error handling necessary in GetCopyStart()?
> Attached the latest code and test patches.
For me the test is still broken:
ok 1 - testlibpqbatch disallowed_in_batch
ok 2 - testlibpqbatch simple_batch
ok 3 - testlibpqbatch multi_batch
ok 4 - testlibpqbatch batch_abort
ok 5 - testlibpqbatch timings
not ok 6 - testlibpqbatch copyfailure
Still broken here. I can see that this patch is having enough
safeguards in PQbatchBegin() and PQsendQueryStart(), but this issue is
really pointing out at something...
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2017-03-30 05:02 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
0 siblings, 1 reply; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-03-30 05:02 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Thu, Mar 30, 2017 at 12:08 PM, Michael Paquier <michael.paquier@gmail.com
> wrote:
> On Wed, Mar 29, 2017 at 12:40 PM, Vaishnavi Prabakaran
> <vaishnaviprabakaran@gmail.com> wrote:
> > Michael Paquier wrote:
> >>Could you as well update src/tools/msvc/vcregress.pl, aka the routine
> >>modulescheck so as this new test is skipped. I am sure that nobody
> >>will scream if this test is not run on Windows, but the buildfarm will
> >>complain if the patch is let in its current state.
> >
> > Thanks for the finding. Hmm, modulescheck will not pick up test_libpq as
> > "subdircheck" will fail for this module(because it does not have
> > "sql"/"expected" folders in it) and hence it will be skipped.
> > But, Modified "Mkvcbuild.pm" to exclude "test_libpq" module, as this test
> > anyways will not be run in Windows and also because it uses linux
> specific
> > APIs(gettimeofday , timersub).
>
> src/port/gettimeofday.c extends things on Windows, and we could
> consider to just drop the timing portion for simplicity (except
> Windows I am not seeing any platform missing timersub). But that's
> just a point of detail. Or we could just drop those tests from the
> final patch, and re-implement them after having some psql-level
> subcommands. That would far better for portability.
>
Yes agree, having tests with psql meta commands will be more easy to
understand also. And, that is one of the reason I separated the patch into
two - code and test . So, yes, we can drop the test patch for now.
>
> > There are 2 cases tested here:
> >
> > 1. Example for the case - Using COPY command in batch mode will trigger
> > failure. (Specified in documentation)
> > Failure can be seen only when processing the copy command's results. That
> > is, after dispatching COPY command, server goes into COPY state , but the
> > client dispatched another query in batch mode.
> >
> > Below error messages belongs to this case :
> > Error status and message got from server due to COPY command failure are
> :
> > PGRES_FATAL_ERROR ERROR: unexpected message type 0x50 during COPY from
> > stdin
> > CONTEXT: COPY batch_demo, line 1
> >
> > message type 0x5a arrived from server while idle
>
> Such messages are really confusing to the user as they refer to the
> protocol going out of sync. I would think that something like "cannot
> process COPY results during a batch processing" would be cleaner.
> Isn't some more error handling necessary in GetCopyStart()?
Hmm, With batch mode, after sending COPY command to server(and server
started processing the query and goes into COPY state) , client does not
immediately read the result , but it keeps sending other queries to the
server. By this time, server already encountered the error
scenario(Receiving different message during COPY state) and sent error
messages. So, when client starts reading the result, already error messages
sent by server are present in socket. I think trying to handle during
result processing is more like masking the server's error message with new
error message and I think it is not appropriate.
>
>
> Attached the latest code and test patches.
>
> For me the test is still broken:
> ok 1 - testlibpqbatch disallowed_in_batch
> ok 2 - testlibpqbatch simple_batch
> ok 3 - testlibpqbatch multi_batch
> ok 4 - testlibpqbatch batch_abort
> ok 5 - testlibpqbatch timings
> not ok 6 - testlibpqbatch copyfailure
>
> Still broken here. I can see that this patch is having enough
> safeguards in PQbatchBegin() and PQsendQueryStart(), but this issue is
> really pointing out at something...
>
>
I will check this one with fresh setup again and I guess that this should
not block the code patch.
Thanks & Regards,
Vaishnavi
Fujitsu Australia.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-03-30 18:12 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
0 siblings, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2017-03-30 18:12 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Craig Ringer <craig@2ndquadrant.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Vaishnavi Prabakaran wrote:
> Hmm, With batch mode, after sending COPY command to server(and server
> started processing the query and goes into COPY state) , client does not
> immediately read the result , but it keeps sending other queries to the
> server. By this time, server already encountered the error
> scenario(Receiving different message during COPY state) and sent error
> messages
IOW, the test intentionally violates the protocol and then all goes wonky
because of that.
That's why I was wondering upthread what's it's supposed to test.
I mean, regression tests are meant to warn against a desirable behavior
being unknowingly changed by new code into an undesirable behavior.
Here we have the undesirable behavior to start with.
What kind of regression could we fear from that?
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2017-03-31 15:03 ` David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: David Steele @ 2017-03-31 15:03 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Craig Ringer <craig@2ndquadrant.com>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Hi,
On 3/30/17 2:12 PM, Daniel Verite wrote:
> Vaishnavi Prabakaran wrote:
>
>> Hmm, With batch mode, after sending COPY command to server(and server
>> started processing the query and goes into COPY state) , client does not
>> immediately read the result , but it keeps sending other queries to the
>> server. By this time, server already encountered the error
>> scenario(Receiving different message during COPY state) and sent error
>> messages
>
> IOW, the test intentionally violates the protocol and then all goes wonky
> because of that.
> That's why I was wondering upthread what's it's supposed to test.
> I mean, regression tests are meant to warn against a desirable behavior
> being unknowingly changed by new code into an undesirable behavior.
> Here we have the undesirable behavior to start with.
> What kind of regression could we fear from that?
The CF has been extended until April 7 but time is still growing short.
Please respond with a new patch by 2017-04-04 00:00 AoE (UTC-12) or this
submission will be marked "Returned with Feedback".
Thanks,
--
-David
david@pgmasters.net
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
@ 2017-04-03 04:10 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-04-03 04:10 UTC (permalink / raw)
To: David Steele <david@pgmasters.net>; +Cc: Craig Ringer <craig@2ndquadrant.com>; Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Sat, Apr 1, 2017 at 2:03 AM, David Steele <david@pgmasters.net> wrote:
> Hi,
>
> On 3/30/17 2:12 PM, Daniel Verite wrote:
>
>> Vaishnavi Prabakaran wrote:
>>
>> Hmm, With batch mode, after sending COPY command to server(and server
>>> started processing the query and goes into COPY state) , client does not
>>> immediately read the result , but it keeps sending other queries to the
>>> server. By this time, server already encountered the error
>>> scenario(Receiving different message during COPY state) and sent error
>>> messages
>>>
>>
>> IOW, the test intentionally violates the protocol and then all goes wonky
>> because of that.
>> That's why I was wondering upthread what's it's supposed to test.
>> I mean, regression tests are meant to warn against a desirable behavior
>> being unknowingly changed by new code into an undesirable behavior.
>> Here we have the undesirable behavior to start with.
>> What kind of regression could we fear from that?
>>
>
Yes, completely agree, demonstrating the undesirable behavior is not needed
as documentation gives enough warning to user.
The test patch is decided not to go in for now, but will be re-implemented
with PSQL commands later. So, during the re-implementation of test patch, I
will remove this test. Thanks .
>
> The CF has been extended until April 7 but time is still growing short.
> Please respond with a new patch by 2017-04-04 00:00 AoE (UTC-12) or this
> submission will be marked "Returned with Feedback".
>
>
Thanks for the information, attached the latest patch resolving one
compilation warning. And, please discard the test patch as it will be
re-implemented later separately.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v9.patch (application/octet-stream, 51.6K) download
---
doc/src/sgml/libpq.sgml | 529 ++++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 594 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 +-
10 files changed, 1211 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..3b7e37b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,527 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchQueueProcess</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4695,6 +5216,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchQueueProcess</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 9d7bb25..a30a032 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -917,6 +917,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 27155f8..bb10b89 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3459,6 +3459,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3470,6 +3489,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3502,6 +3522,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9decd53..b0f5ce7 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,297 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2217,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2422,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2623,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2641,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2663,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2688,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2569,6 +3090,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index b8ec341..33f212f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -384,6 +406,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -693,6 +726,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-04-03 21:05 ` Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Andres Freund @ 2017-04-03 21:05 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: David Steele <david@pgmasters.net>; Craig Ringer <craig@2ndquadrant.com>; Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 2017-04-03 14:10:47 +1000, Vaishnavi Prabakaran wrote:
> > The CF has been extended until April 7 but time is still growing short.
> > Please respond with a new patch by 2017-04-04 00:00 AoE (UTC-12) or this
> > submission will be marked "Returned with Feedback".
> >
> >
> Thanks for the information, attached the latest patch resolving one
> compilation warning. And, please discard the test patch as it will be
> re-implemented later separately.
Hm. If the tests aren't ready yet, it seems we'll have to move this to
the next CF.
> + <sect1 id="libpq-batch-mode">
> + <title>Batch mode and query pipelining</title>
> +
> + <indexterm zone="libpq-batch-mode">
> + <primary>libpq</primary>
> + <secondary>batch mode</secondary>
> + </indexterm>
> +
> + <indexterm zone="libpq-batch-mode">
> + <primary>libpq</primary>
> + <secondary>pipelining</secondary>
> + </indexterm>
> +
> + <para>
> + <application>libpq</application> supports queueing up multiple queries into
> + a pipeline to be executed as a batch on the server. Batching queries allows
> + applications to avoid a client/server round-trip after each query to get
> + the results before issuing the next query.
> + </para>
"queueing .. into a pipeline" sounds weird to me - but I'm not a native
speaker. Also batching != pipelining.
> + <sect2>
> + <title>When to use batching</title>
> +
> + <para>
> + Much like asynchronous query mode, there is no performance disadvantage to
> + using batching and pipelining. It increases client application complexity
> + and extra caution is required to prevent client/server deadlocks but
> + offers considerable performance improvements.
> + </para>
s/offers/can sometimes offer/
> + <sect2 id="libpq-batch-using">
> + <title>Using batch mode</title>
> +
> + <para>
> + To issue batches the application must switch
> + <application>libpq</application> into batch mode.
s/libpq/a connection/?
> Enter batch mode with <link
> + linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
> + whether batch mode is active with <link
> + linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
> + linkend="libpq-async">asynchronous operations</link> are permitted, and
> + <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
> + (The restriction on <literal>COPY</literal> is an implementation
> + limit; the PostgreSQL protocol and server can support batched
> <literal>COPY</literal>).
Hm, I'm unconvinced that that's a useful parenthetical in the libpq
docs.
> + <para>
> + The client uses libpq's asynchronous query functions to dispatch work,
> + marking the end of each batch with <function>PQbatchQueueSync</function>.
> + Concurrently, it uses <function>PQgetResult</function> and
> + <function>PQbatchQueueProcess</function> to get results.
"Concurrently" imo is a dangerous word, somewhat implying multi-threading.
> + <note>
> + <para>
> + It is best to use batch mode with <application>libpq</application> in
> + <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
> + blocking mode it is possible for a client/server deadlock to occur. The
> + client will block trying to send queries to the server, but the server will
> + block trying to send results from queries it has already processed to the
> + client. This only occurs when the client sends enough queries to fill its
> + output buffer and the server's receive buffer before switching to
> + processing input from the server, but it's hard to predict exactly when
> + that'll happen so it's best to always use non-blocking mode.
> + </para>
> + </note>
Such deadlocks are possible just as well with non-blocking mode, unless
one can avoid sending queries and switching to receiving results anytime
in the application code.
> + <para>
> + Batched operations will be executed by the server in the order the client
> + sends them. The server will send the results in the order the statements
> + executed. The server may begin executing the batch before all commands
> + in the batch are queued and the end of batch command is sent. If any
> + statement encounters an error the server aborts the current transaction and
> + skips processing the rest of the batch. Query processing resumes after the
> + end of the failed batch.
> + </para>
What if a batch contains transaction boundaries?
> + <sect3 id="libpq-batch-results">
> + <title>Processing results</title>
> +
> + <para>
> + The client <link linkend="libpq-batch-interleave">interleaves result
> + processing with sending batch queries</link>, or for small batches may
> + process all results after sending the whole batch.
> + </para>
That's a very long <link> text, is it not?
> + <para>
> + To get the result of the first batch entry the client must call <link
> + linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
What does 'QueueProcess' mean? Shouldn't it be 'ProcessQueue'? You're
not enquing a process or processing, right?
> + <function>PQgetResult</function> and handle the results until
> + <function>PQgetResult</function> returns null (or would return null if
> + called).
What is that parenthetical referring to? IIRC we don't provide any
external way to determine PQgetResult would return NULL.
Have you checked how this API works with PQsetSingleRowMode()?
> + <para>
> + To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
> + successful call of <function>PQbatchQueueProcess</function>. This mode selection is effective
> + only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
> + </function>, refer to <xref linkend="libpq-single-row-mode">.
Hah ;).
> + <para>
> + The client must not assume that work is committed when it
> + <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
> + corresponding result is received to confirm the commit is complete.
> + Because errors arrive asynchronously the application needs to be able to
> + restart from the last <emphasis>received</emphasis> committed change and
> + resend work done after that point if something goes wrong.
> + </para>
That seems like a batch independent thing, right? If so, maybe make it
a <note>?
> + <listitem>
> + <para>
> + Causes a connection to enter batch mode if it is currently idle or
> + already in batch mode.
> +
> +<synopsis>
> +int PQbatchBegin(PGconn *conn);
> +</synopsis>
> +
> + </para>
> + <para>
> + Returns 1 for success. Returns 0 and has no
> + effect if the connection is not currently idle, i.e. it has a result
> + ready, is waiting for more input from the server, etc. This function
> + does not actually send anything to the server, it just changes the
> + <application>libpq</application> connection state.
> +
> + </para>
> + </listitem>
> + </varlistentry>
That function name sounds a bit too much like it'd be relevant for a
single batch, not something that can send many batches. enterBatchMode?
> + <varlistentry id="libpq-PQbatchEnd">
> + <term>
> + <function>PQbatchEnd</function>
> + <indexterm>
> + <primary>PQbatchEnd</primary>
> + </indexterm>
> + </term>
> +
> + <listitem>
> + <para>
> + Causes a connection to exit batch mode if it is currently in batch mode
> + with an empty queue and no pending results.
> +<synopsis>
> +int PQbatchEnd(PGconn *conn);
> +</synopsis>
> + </para>
> + <para>Returns 1 for success.
> + Returns 1 and takes no action if not in batch mode. If the connection has
> + pending batch items in the queue for reading with
> + <function>PQbatchQueueProcess</function>, the current statement isn't finished
> + processing or there are results pending for collection with
> + <function>PQgetResult</function>, returns 0 and does nothing.
> +
> + </para>
> + </listitem>
> + </varlistentry>
""
> + <varlistentry id="libpq-PQbatchQueueSync">
> + <term>
> + <function>PQbatchQueueSync</function>
> + <function>PQbatchQueueProcess</function>
As said above, I'm not a fan of these, because it sounds like you're
queueing a sync/process.
> /* ----------------
> @@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
> conn->next_result = conn->result;
> conn->result = res;
> /* And mark the result ready to return */
> - conn->asyncStatus = PGASYNC_READY;
> + conn->asyncStatus = PGASYNC_READY_MORE;
> }
Uhm, isn't that an API/ABI breakage issue?
> /*
> - * Common startup code for PQsendQuery and sibling routines
> + * PQmakePipelinedCommand
> + * Get a new command queue entry, allocating it if required. Doesn't add it to
> + * the tail of the queue yet, use PQappendPipelinedCommand once the command has
> + * been written for that. If a command fails once it's called this, it should
> + * use PQrecyclePipelinedCommand to put it on the freelist or release it.
"command fails once it's called this"?
> +/*
> + * PQrecyclePipelinedCommand
> + * Push a command queue entry onto the freelist. It must be a dangling entry
> + * with null next pointer and not referenced by any other entry's next pointer.
> + */
> +static void
> +PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
> +{
> + if (entry == NULL)
> + return;
> + if (entry->next != NULL)
> + {
> + fprintf(stderr, "tried to recycle non-dangling command queue entry");
> + abort();
Needs a libpq_gettext()?
> +/*
> + * PQbatchEnd
> + * End batch mode and return to normal command mode.
> + *
> + * Has no effect unless the client has processed all results
> + * from all outstanding batches and the connection is idle.
> + *
> + * Returns true if batch mode ended.
> + */
> +int
> +PQbatchEnd(PGconn *conn)
> +{
> + if (!conn)
> + return false;
> +
> + if (conn->batch_status == PQBATCH_MODE_OFF)
> + return true;
> +
> + switch (conn->asyncStatus)
> + {
> + case PGASYNC_IDLE:
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext_noop("internal error, IDLE in batch mode"));
> + break;
> + case PGASYNC_COPY_IN:
> + case PGASYNC_COPY_OUT:
> + case PGASYNC_COPY_BOTH:
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext_noop("internal error, COPY in batch mode"));
> + break;
Why aren't you returning false here,
> + case PGASYNC_READY:
> + case PGASYNC_READY_MORE:
> + case PGASYNC_BUSY:
> + /* can't end batch while busy */
> + return false;
but are here?
> + case PGASYNC_QUEUED:
> + break;
> + }
> +
> +int
> +PQbatchQueueSync(PGconn *conn)
> +{
> + PGcommandQueueEntry *entry;
> +
> + if (!conn)
> + return false;
> +
> + if (conn->batch_status == PQBATCH_MODE_OFF)
> + return false;
> +
> + switch (conn->asyncStatus)
> + {
> + case PGASYNC_IDLE:
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext_noop("internal error, IDLE in batch mode"));
> + break;
> + case PGASYNC_COPY_IN:
> + case PGASYNC_COPY_OUT:
> + case PGASYNC_COPY_BOTH:
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext_noop("internal error, COPY in batch mode"));
> + break;
> + case PGASYNC_READY:
> + case PGASYNC_READY_MORE:
> + case PGASYNC_BUSY:
> + case PGASYNC_QUEUED:
> + /* can send sync to end this batch of cmds */
> + break;
> + }
Uhm, what is that switch actually achieving? We're not returning an
error code, so ...?
> + /* Should try to flush immediately if there's room */
> + PQflush(conn);
"room"?
Also, don't we need to process PQflush's return value?
> +/*
> + * PQbatchQueueProcess
> + * In batch mode, start processing the next query in the queue.
> + *
> + * Returns true if the next query was popped from the queue and can
> + * be processed by PQconsumeInput, PQgetResult, etc.
> + *
> + * Returns false if the current query isn't done yet, the connection
> + * is not in a batch, or there are no more queries to process.
> + */
> +int
> +PQbatchQueueProcess(PGconn *conn)
> +{
> + PGcommandQueueEntry *next_query;
> +
> + if (!conn)
> + return false;
> +
> + if (conn->batch_status == PQBATCH_MODE_OFF)
> + return false;
> +
> + switch (conn->asyncStatus)
> + {
> + case PGASYNC_COPY_IN:
> + case PGASYNC_COPY_OUT:
> + case PGASYNC_COPY_BOTH:
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext_noop("internal error, COPY in batch mode"));
> + break;
> + case PGASYNC_READY:
> + case PGASYNC_READY_MORE:
> + case PGASYNC_BUSY:
> + /* client still has to process current query or results */
> + return false;
> + break;
> + case PGASYNC_IDLE:
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext_noop("internal error, IDLE in batch mode"));
> + break;
> + case PGASYNC_QUEUED:
> + /* next query please */
> + break;
> + }
Once more, I'm very unconvinced by the switch. Unless you do anything
with the errors, this seems pointless.
> + if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
> + {
> + /*
> + * In an aborted batch we don't get anything from the server for each
> + * result; we're just discarding input until we get to the next sync
> + * from the server. The client needs to know its queries got aborted
> + * so we create a fake PGresult to return immediately from
> + * PQgetResult.
> + */
> + conn->result = PQmakeEmptyPGresult(conn,
> + PGRES_BATCH_ABORTED);
> + if (!conn->result)
> + {
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext("out of memory"));
> + pqSaveErrorResult(conn);
> + }
> + conn->asyncStatus = PGASYNC_READY;
So we still return true in the OOM case?
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-04-03 23:24 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-04-03 23:24 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: David Steele <david@pgmasters.net>; Craig Ringer <craig@2ndquadrant.com>; Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Hi,
On Tue, Apr 4, 2017 at 7:05 AM, Andres Freund <andres@anarazel.de> wrote:
> On 2017-04-03 14:10:47 +1000, Vaishnavi Prabakaran wrote:
> > > The CF has been extended until April 7 but time is still growing short.
> > > Please respond with a new patch by 2017-04-04 00:00 AoE (UTC-12) or
> this
> > > submission will be marked "Returned with Feedback".
> > >
> > >
> > Thanks for the information, attached the latest patch resolving one
> > compilation warning. And, please discard the test patch as it will be
> > re-implemented later separately.
>
> Hm. If the tests aren't ready yet, it seems we'll have to move this to
> the next CF.
>
>
Thanks for your review and I will address your review comments and send the
newer version of patch shortly.
Just quickly, Is it not ok to consider only the code patch for this CF
without test patch?
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-04-03 23:26 ` Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Andres Freund @ 2017-04-03 23:26 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: David Steele <david@pgmasters.net>; Craig Ringer <craig@2ndquadrant.com>; Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 2017-04-04 09:24:23 +1000, Vaishnavi Prabakaran wrote:
> Hi,
> On Tue, Apr 4, 2017 at 7:05 AM, Andres Freund <andres@anarazel.de> wrote:
>
> > On 2017-04-03 14:10:47 +1000, Vaishnavi Prabakaran wrote:
> > > > The CF has been extended until April 7 but time is still growing short.
> > > > Please respond with a new patch by 2017-04-04 00:00 AoE (UTC-12) or
> > this
> > > > submission will be marked "Returned with Feedback".
> > > >
> > > >
> > > Thanks for the information, attached the latest patch resolving one
> > > compilation warning. And, please discard the test patch as it will be
> > > re-implemented later separately.
> >
> > Hm. If the tests aren't ready yet, it seems we'll have to move this to
> > the next CF.
> >
> >
> Thanks for your review and I will address your review comments and send the
> newer version of patch shortly.
Cool.
> Just quickly, Is it not ok to consider only the code patch for this CF
> without test patch?
I'd say no, it's not acceptable. This is too much new code for it not
to be tested.
Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-04-03 23:57 ` Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 182+ messages in thread
From: Michael Paquier @ 2017-04-03 23:57 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; David Steele <david@pgmasters.net>; Craig Ringer <craig@2ndquadrant.com>; Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Tue, Apr 4, 2017 at 8:26 AM, Andres Freund <andres@anarazel.de> wrote:
> On 2017-04-04 09:24:23 +1000, Vaishnavi Prabakaran wrote:
>> Just quickly, Is it not ok to consider only the code patch for this CF
>> without test patch?
>
> I'd say no, it's not acceptable. This is too much new code for it not
> to be tested.
Doesn't it depend actually? I would think that the final patch may not
include all the tests implemented if:
- The thread on which a patch has been developed had a set of tests
done and posted with it.
- Including them does not make sense if we have a way to run those
tests more efficiently. Sometimes a bunch of benchmarks or tests are
run on a patch bu for the final result keeping them around does not
make much sense.
In the case of this patch, it seems to me that we would have a far
better portable set of tests if we had a dedicated set of subcommands
available at psql level, particularly for Windows/MSVC. If that's a
requirement for this patch so let it be. I am not saying that tests
are not necessary. They are of course, but in this case having a bit
more infrastructure would be more be more helpful for users and the
tests themselves.
Note that I won't complain either if this set of C tests are included
at the end.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2017-04-04 04:52 ` Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Andres Freund @ 2017-04-04 04:52 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; David Steele <david@pgmasters.net>; Craig Ringer <craig@2ndquadrant.com>; Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 2017-04-04 08:57:33 +0900, Michael Paquier wrote:
> On Tue, Apr 4, 2017 at 8:26 AM, Andres Freund <andres@anarazel.de> wrote:
> > On 2017-04-04 09:24:23 +1000, Vaishnavi Prabakaran wrote:
> >> Just quickly, Is it not ok to consider only the code patch for this CF
> >> without test patch?
> >
> > I'd say no, it's not acceptable. This is too much new code for it not
> > to be tested.
>
> Doesn't it depend actually?
Well, I didn't make a general statement, I made one about this patch.
And this would add a significant bunch of untested code, and it'll likely
take years till it gets decent coverage outside.
> In the case of this patch, it seems to me that we would have a far
> better portable set of tests if we had a dedicated set of subcommands
> available at psql level, particularly for Windows/MSVC.
That's a really large scope creep imo. Adding a bunch of user-facing
psql stuff doesn't compare in complexity to running a test across
platforms. We can just do that from regess.c or such, if that ends up
being a problem..
> If that's a requirement for this patch so let it be. I am not saying that tests
> are not necessary. They are of course, but in this case having a bit
> more infrastructure would be more be more helpful for users and the
> tests themselves.
I'm not following.
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-04-05 07:00 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-04-05 07:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Craig Ringer <craig@2ndquadrant.com>; Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Andres Freund <andres@anarazel.de> wrote:
> > +
> > + <para>
> > + <application>libpq</application> supports queueing up multiple
> queries into
> > + a pipeline to be executed as a batch on the server. Batching queries
> allows
> > + applications to avoid a client/server round-trip after each query to
> get
> > + the results before issuing the next query.
> > + </para>
>
> "queueing .. into a pipeline" sounds weird to me - but I'm not a native
> speaker. Also batching != pipelining.
Re-phrased the sentence as "libpq supports queries to be queued up in
batches and pipeline them to the server, where it will be executed as a
batch" . Pipelining queries allows applications to avoid a client/server
round-trip after each query to get the results before issuing the next
query.
>
> > + <sect2>
> > + <title>When to use batching</title>
> > +
> > + <para>
> > + Much like asynchronous query mode, there is no performance
> disadvantage to
> > + using batching and pipelining. It increases client application
> complexity
> > + and extra caution is required to prevent client/server deadlocks but
> > + offers considerable performance improvements.
> > + </para>
>
> s/offers/can sometimes offer/
>
>
Corrected.
>
> > + <sect2 id="libpq-batch-using">
> > + <title>Using batch mode</title>
> > +
> > + <para>
> > + To issue batches the application must switch
> > + <application>libpq</application> into batch mode.
>
> s/libpq/a connection/?
>
>
Corrected.
>
> > Enter batch mode with <link
> > + linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link>
> or test
> > + whether batch mode is active with <link
> > + linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>.
> In batch mode only <link
> > + linkend="libpq-async">asynchronous operations</link> are
> permitted, and
> > + <literal>COPY</literal> is not recommended as it most likely will
> trigger failure in batch processing.
> > + (The restriction on <literal>COPY</literal> is an implementation
> > + limit; the PostgreSQL protocol and server can support batched
> > <literal>COPY</literal>).
>
> Hm, I'm unconvinced that that's a useful parenthetical in the libpq
> docs.
>
>
Removed the parenthesis as the line before gives enough warning.
>
> > + <para>
> > + The client uses libpq's asynchronous query functions to dispatch
> work,
> > + marking the end of each batch with <function>PQbatchQueueSync</
> function>.
> > + Concurrently, it uses <function>PQgetResult</function> and
> > + <function>PQbatchQueueProcess</function> to get results.
>
> "Concurrently" imo is a dangerous word, somewhat implying multi-threading.
>
>
Corrected by replacing "concurrently" with "And to get results"
>
> > + <note>
> > + <para>
> > + It is best to use batch mode with <application>libpq</application>
> in
> > + <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>.
> If used in
> > + blocking mode it is possible for a client/server deadlock to
> occur. The
> > + client will block trying to send queries to the server, but the
> server will
> > + block trying to send results from queries it has already processed
> to the
> > + client. This only occurs when the client sends enough queries to
> fill its
> > + output buffer and the server's receive buffer before switching to
> > + processing input from the server, but it's hard to predict exactly
> when
> > + that'll happen so it's best to always use non-blocking mode.
> > + </para>
> > + </note>
>
> Such deadlocks are possible just as well with non-blocking mode, unless
> one can avoid sending queries and switching to receiving results anytime
> in the application code.
>
>
True, and in the documentation , how to interleave result processing and
query dispatch is given.
>
> > + <para>
> > + Batched operations will be executed by the server in the order the
> client
> > + sends them. The server will send the results in the order the
> statements
> > + executed. The server may begin executing the batch before all
> commands
> > + in the batch are queued and the end of batch command is sent. If
> any
> > + statement encounters an error the server aborts the current
> transaction and
> > + skips processing the rest of the batch. Query processing resumes
> after the
> > + end of the failed batch.
> > + </para>
>
> What if a batch contains transaction boundaries?
>
>
Regardless of transaction boundaries, failed command will make the entire
batch aborted and no commands will be processed in server until Batch sync
is sent. And, if the command inside the transaction fails, then the batch
will be aborted, but transaction will still be open in the server. Client
explicitly needs to close the transaction. This is explained further down
in the documentation section "Error handling" , paragraph starting with "If
the batch used an implicit transaction.." and "Note" in that section gives
warning too.
>
> > + <sect3 id="libpq-batch-results">
> > + <title>Processing results</title>
> > +
> > + <para>
> > + The client <link linkend="libpq-batch-interleave">interleaves
> result
> > + processing with sending batch queries</link>, or for small batches
> may
> > + process all results after sending the whole batch.
> > + </para>
>
> That's a very long <link> text, is it not?
>
>
Corrected.
>
> > + <para>
> > + To get the result of the first batch entry the client must call
> <link
> > + linkend="libpq-PQbatchQueueProcess"><
> function>PQbatchQueueProcess</function></link>. It must then call
>
> What does 'QueueProcess' mean? Shouldn't it be 'ProcessQueue'? You're
> not enquing a process or processing, right?
>
>
Changed to PQbatchProcessQueue.
>
> > + <function>PQgetResult</function> and handle the results until
> > + <function>PQgetResult</function> returns null (or would return
> null if
> > + called).
>
> What is that parenthetical referring to? IIRC we don't provide any
> external way to determine PQgetResult would return NULL.
>
>
True, it is just application's extra knowledge. Removed the parenthesis.
> > + <para>
> > + The client must not assume that work is committed when it
> > + <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when
> the
> > + corresponding result is received to confirm the commit is complete.
> > + Because errors arrive asynchronously the application needs to be
> able to
> > + restart from the last <emphasis>received</emphasis> committed
> change and
> > + resend work done after that point if something goes wrong.
> > + </para>
>
> That seems like a batch independent thing, right? If so, maybe make it
> a <note>?
>
>
Yes, made it as Note.
> > +
> > +<synopsis>
> > +int PQbatchBegin(PGconn *conn);
> ...
>
> That function name sounds a bit too much like it'd be relevant for a
> single batch, not something that can send many batches. enterBatchMode?
>
> > +int PQbatchEnd(PGconn *conn);
> ...
> ""
>
>
Changed the function names to PQenterBatchMode and PQexitBatchMode.
>
> > + <varlistentry id="libpq-PQbatchQueueSync">
> > + <term>
> > + <function>PQbatchQueueSync</function>
> > + <function>PQbatchQueueProcess</function>
>
> As said above, I'm not a fan of these, because it sounds like you're
> queueing a sync/process.
>
>
Changed it to PQbatchSyncQueue.
>
> > /* ----------------
> > @@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
> > conn->next_result = conn->result;
> > conn->result = res;
> > /* And mark the result ready to return */
> > - conn->asyncStatus = PGASYNC_READY;
> > + conn->asyncStatus = PGASYNC_READY_MORE;
> > }
>
> Uhm, isn't that an API/ABI breakage issue?
>
>
I think no, asynstatus is in libpq-int.h which is meant to be used only by
frontend libpq, applications should use libpq-fe.h . Also, we don't have
any API to return the async status of connection to application. So, I
think it will not break any existing applications.
>
> > /*
> > - * Common startup code for PQsendQuery and sibling routines
> > + * PQmakePipelinedCommand
> > + * Get a new command queue entry, allocating it if required. Doesn't
> add it to
> > + * the tail of the queue yet, use PQappendPipelinedCommand once the
> command has
> > + * been written for that. If a command fails once it's called this,
> it should
> > + * use PQrecyclePipelinedCommand to put it on the freelist or release
> it.
>
> "command fails once it's called this"?
>
>
Corrected.- "Command sending fails"
>
> > +/*
> > + * PQrecyclePipelinedCommand
> > + * Push a command queue entry onto the freelist. It must be a
> dangling entry
> > + * with null next pointer and not referenced by any other entry's
> next pointer.
> > + */
> > +static void
> > +PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
> > +{
> > + if (entry == NULL)
> > + return;
> > + if (entry->next != NULL)
> > + {
> > + fprintf(stderr, "tried to recycle non-dangling command
> queue entry");
> > + abort();
>
> Needs a libpq_gettext()?
>
>
>
Corrected.
>
> > +/*
> > + * PQbatchEnd
> > + * End batch mode and return to normal command mode.
> > + *
> > + * Has no effect unless the client has processed all results
> > + * from all outstanding batches and the connection is idle.
> > + *
> > + * Returns true if batch mode ended.
> > + */
> > +int
> > +PQbatchEnd(PGconn *conn)
> > +{
> > + if (!conn)
> > + return false;
> > +
> > + if (conn->batch_status == PQBATCH_MODE_OFF)
> > + return true;
> > +
> > + switch (conn->asyncStatus)
> > + {
> > + case PGASYNC_IDLE:
> > + printfPQExpBuffer(&conn->errorMessage,
> > + libpq_gettext_noop("internal error,
> IDLE in batch mode"));
> > + break;
> > + case PGASYNC_COPY_IN:
> > + case PGASYNC_COPY_OUT:
> > + case PGASYNC_COPY_BOTH:
> > + printfPQExpBuffer(&conn->errorMessage,
> > + libpq_gettext_noop("internal error,
> COPY in batch mode"));
> > + break;
>
> Why aren't you returning false here,
Using copy command in batch mode is the error case, so instead of giving
opportunity to the application to finish the error scenario, we set the
error message and allows it to quit the batch mode.
>
> > + case PGASYNC_READY:
> > + case PGASYNC_READY_MORE:
> > + case PGASYNC_BUSY:
> > + /* can't end batch while busy */
> > + return false;
>
> but are here?
Application gets a chance to finish the result processing and exit the
batch mode.
> > +int
> > +PQbatchQueueSync(PGconn *conn)
> > +{
> > + PGcommandQueueEntry *entry;
> > +
> > + if (!conn)
> > + return false;
> > +
> > + if (conn->batch_status == PQBATCH_MODE_OFF)
> > + return false;
> > +
> > + switch (conn->asyncStatus)
> > + {
> > + case PGASYNC_IDLE:
> > + printfPQExpBuffer(&conn->errorMessage,
> > + libpq_gettext_noop("internal error,
> IDLE in batch mode"));
> > + break;
> > + case PGASYNC_COPY_IN:
> > + case PGASYNC_COPY_OUT:
> > + case PGASYNC_COPY_BOTH:
> > + printfPQExpBuffer(&conn->errorMessage,
> > + libpq_gettext_noop("internal error,
> COPY in batch mode"));
> > + break;
> > + case PGASYNC_READY:
> > + case PGASYNC_READY_MORE:
> > + case PGASYNC_BUSY:
> > + case PGASYNC_QUEUED:
> > + /* can send sync to end this batch of cmds */
> > + break;
> > + }
>
> Uhm, what is that switch actually achieving? We're not returning an
> error code, so ...?
These errors are marked here, so that the application later when reading
the results for batch queries, will have more information about what went
wrong. This is just an additional information to user only, but doesn't
stop the error scenario.
>
>
> > + /* Should try to flush immediately if there's room */
> > + PQflush(conn);
>
> "room"?
>
> Also, don't we need to process PQflush's return value?
>
>
Corrected both .
>
> > +/*
> > + * PQbatchQueueProcess
> > + * In batch mode, start processing the next query in the queue.
> > + *
> > + * Returns true if the next query was popped from the queue and can
> > + * be processed by PQconsumeInput, PQgetResult, etc.
> > + *
> > + * Returns false if the current query isn't done yet, the connection
> > + * is not in a batch, or there are no more queries to process.
> > + */
> > +int
> > +PQbatchQueueProcess(PGconn *conn)
> > +{
> > + PGcommandQueueEntry *next_query;
> > +
> > + if (!conn)
> > + return false;
> > +
> > + if (conn->batch_status == PQBATCH_MODE_OFF)
> > + return false;
> > +
> > + switch (conn->asyncStatus)
> > + {
> > + case PGASYNC_COPY_IN:
> > + case PGASYNC_COPY_OUT:
> > + case PGASYNC_COPY_BOTH:
> > + printfPQExpBuffer(&conn->errorMessage,
> > + libpq_gettext_noop("internal error,
> COPY in batch mode"));
> > + break;
> > + case PGASYNC_READY:
> > + case PGASYNC_READY_MORE:
> > + case PGASYNC_BUSY:
> > + /* client still has to process current query or
> results */
> > + return false;
> > + break;
> > + case PGASYNC_IDLE:
> > + printfPQExpBuffer(&conn->errorMessage,
> > + libpq_gettext_noop("internal error,
> IDLE in batch mode"));
> > + break;
> > + case PGASYNC_QUEUED:
> > + /* next query please */
> > + break;
> > + }
>
> Once more, I'm very unconvinced by the switch. Unless you do anything
> with the errors, this seems pointless.
Same answer as above switch question.
>
>
> > + if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass
> != PGQUERY_SYNC)
> > + {
> > + /*
> > + * In an aborted batch we don't get anything from the
> server for each
> > + * result; we're just discarding input until we get to the
> next sync
> > + * from the server. The client needs to know its queries
> got aborted
> > + * so we create a fake PGresult to return immediately from
> > + * PQgetResult.
> > + */
> > + conn->result = PQmakeEmptyPGresult(conn,
> > +
> PGRES_BATCH_ABORTED);
> > + if (!conn->result)
> > + {
> > + printfPQExpBuffer(&conn->errorMessage,
> > +
> libpq_gettext("out of memory"));
> > + pqSaveErrorResult(conn);
> > + }
> > + conn->asyncStatus = PGASYNC_READY;
>
> So we still return true in the OOM case?
>
>
Corrected to return false, as further calls to this function will also
fail. And , application as well can check for this failure reason from
PQgetResult.
Regarding test patch, I have corrected the test suite after David Steele's
comments.
Also, I would like to mention that a companion patch was submitted by David
Steele up-thread.
Attached the latest code and test patch.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments: download all (2 files, .tar.gz)
0001-Pipelining-batch-support-for-libpq-code-v10.patch (application/octet-stream, 51.7K) download
---
doc/src/sgml/libpq.sgml | 528 ++++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 599 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 +-
10 files changed, 1215 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..8aec588 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,526 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSyncQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSyncQueue"><function>PQbatchSyncQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSyncQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSyncQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSyncQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSyncQueue">
+ <term>
+ <function>PQbatchSyncQueue</function>
+ <indexterm>
+ <primary>PQbatchSyncQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSyncQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchProcessQueue</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4695,6 +5215,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 9d7bb25..a30a032 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -917,6 +917,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..3d7ec85 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQenterBatchMode 173
+PQexitBatchMode 174
+PQbatchSyncQueue 175
+PQbatchProcessQueue 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index d79a1cf..ea33543 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3459,6 +3459,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3470,6 +3489,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3502,6 +3522,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9decd53..2a3928b 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,302 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSyncQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return false;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2222,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2427,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2628,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2646,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2668,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2693,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2569,6 +3095,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..04aaded 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSyncQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index b8ec341..33f212f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -384,6 +406,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -693,6 +726,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
0002-Pipelining-batch-support-for-libpq-test-v6.patch (application/octet-stream, 39.5K) download
---
src/test/modules/Makefile | 1 +
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 25 +
src/test/modules/test_libpq/testlibpqbatch.c | 1452 ++++++++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 2 +-
7 files changed, 1510 insertions(+), 1 deletion(-)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 3ce9904..310d3ba 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -11,6 +11,7 @@ SUBDIRS = \
snapshot_too_old \
test_ddl_deparse \
test_extensions \
+ test_libpq \
test_parser \
test_pg_dump \
test_rls_hooks \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..25f9ff0
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,25 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..88d6d60
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1452 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchSyncQueue(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchProcessQueue when there might still be pending results */
+ if (PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSyncQueue(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSyncQueue(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSyncQueue(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSyncQueue(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSyncQueue(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQenterBatchMode(conn);
+
+ for (i=0; i < 3; i++) {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchSyncQueue(conn);
+
+ i=0;
+ while (PQbatchProcessQueue(conn))
+ {
+ int isSingleTuple = 0;
+ /* Set single row mode for only first 3 SELECT queries */
+ if(i < 3)
+ r = PQsetSingleRowMode(conn);
+ if (r!=1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ if(!isSingleTuple)
+ {
+ fprintf(stderr, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ goto fail;
+ }
+ isSingleTuple=0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ }
+ i++;
+ }
+ PQexitBatchMode(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if(run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index ba1bf6d..4a2a4dd 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -46,7 +46,7 @@ my @contrib_excludes = (
'ltree_plpython', 'pgcrypto',
'sepgsql', 'brin',
'test_extensions', 'test_pg_dump',
- 'snapshot_too_old');
+ 'snapshot_too_old', 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
--
2.7.4.windows.1
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-04-05 22:45 ` Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 182+ messages in thread
From: Andres Freund @ 2017-04-05 22:45 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Craig Ringer <craig@2ndquadrant.com>; Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Hi,
On 2017-04-05 17:00:42 +1000, Vaishnavi Prabakaran wrote:
> Regarding test patch, I have corrected the test suite after David Steele's
> comments.
> Also, I would like to mention that a companion patch was submitted by David
> Steele up-thread.
>
> Attached the latest code and test patch.
My impression is that this'll need a couple more rounds of review. Given
that this'll establish API we'll pretty much ever going to be able to
change/remove, I think it'd be a bad idea to rush this into v10.
Therefore I propose moving this to the next CF.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-06-19 22:49 ` Andres Freund <andres@anarazel.de>
2017-06-20 00:09 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-06-20 01:43 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
0 siblings, 3 replies; 182+ messages in thread
From: Andres Freund @ 2017-06-19 22:49 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Craig Ringer <craig@2ndquadrant.com>; Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 2017-04-05 15:45:26 -0700, Andres Freund wrote:
> Hi,
>
> On 2017-04-05 17:00:42 +1000, Vaishnavi Prabakaran wrote:
> > Regarding test patch, I have corrected the test suite after David Steele's
> > comments.
> > Also, I would like to mention that a companion patch was submitted by David
> > Steele up-thread.
> >
> > Attached the latest code and test patch.
>
> My impression is that this'll need a couple more rounds of review. Given
> that this'll establish API we'll pretty much ever going to be able to
> change/remove, I think it'd be a bad idea to rush this into v10.
> Therefore I propose moving this to the next CF.
Craig, Vaishnavi, everyone else: Are you planning to continue to work on
this for v11? I'm willing to do another round, but only if it's
worthwhile.
FWIW, I still think this needs a pgbench or similar example integration,
so we can actually properly measure the benefits.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-06-20 00:09 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2 siblings, 0 replies; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-06-20 00:09 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Craig Ringer <craig@2ndquadrant.com>; Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Tue, Jun 20, 2017 at 8:49 AM, Andres Freund <andres@anarazel.de> wrote:
> On 2017-04-05 15:45:26 -0700, Andres Freund wrote:
> > Hi,
> >
> > On 2017-04-05 17:00:42 +1000, Vaishnavi Prabakaran wrote:
> > > Regarding test patch, I have corrected the test suite after David
> Steele's
> > > comments.
> > > Also, I would like to mention that a companion patch was submitted by
> David
> > > Steele up-thread.
> > >
> > > Attached the latest code and test patch.
> >
> > My impression is that this'll need a couple more rounds of review. Given
> > that this'll establish API we'll pretty much ever going to be able to
> > change/remove, I think it'd be a bad idea to rush this into v10.
> > Therefore I propose moving this to the next CF.
>
> Craig, Vaishnavi, everyone else: Are you planning to continue to work on
> this for v11? I'm willing to do another round, but only if it's
> worthwhile.
>
Yes, am willing to continue working on this patch for v11.
>
> FWIW, I still think this needs a pgbench or similar example integration,
> so we can actually properly measure the benefits.
>
I will investigate on this further.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-06-20 01:43 ` Craig Ringer <craig@2ndquadrant.com>
2017-06-20 02:11 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2017-06-20 01:43 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 20 June 2017 at 06:49, Andres Freund <andres@anarazel.de> wrote:
> On 2017-04-05 15:45:26 -0700, Andres Freund wrote:
>> Hi,
>>
>> On 2017-04-05 17:00:42 +1000, Vaishnavi Prabakaran wrote:
>> > Regarding test patch, I have corrected the test suite after David Steele's
>> > comments.
>> > Also, I would like to mention that a companion patch was submitted by David
>> > Steele up-thread.
>> >
>> > Attached the latest code and test patch.
>>
>> My impression is that this'll need a couple more rounds of review. Given
>> that this'll establish API we'll pretty much ever going to be able to
>> change/remove, I think it'd be a bad idea to rush this into v10.
>> Therefore I propose moving this to the next CF.
>
> Craig, Vaishnavi, everyone else: Are you planning to continue to work on
> this for v11? I'm willing to do another round, but only if it's
> worthwhile.
I'm happy to work on review, and will try to make some time, but have
to focus primarily on logical rep infrastructure. This patch was a
proof of concept and fun hack for me and while I'm glad folks are
interested, it's not something I can dedicate much time to. Especially
with a 6-week-old baby now....
> FWIW, I still think this needs a pgbench or similar example integration,
> so we can actually properly measure the benefits.
I agree. I originally wanted to patch psql, but it's pretty intrusive.
pgbench is likely a better target. Also pg_restore.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 01:43 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2017-06-20 02:11 ` Michael Paquier <michael.paquier@gmail.com>
0 siblings, 0 replies; 182+ messages in thread
From: Michael Paquier @ 2017-06-20 02:11 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Andres Freund <andres@anarazel.de>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; David Steele <david@pgmasters.net>; Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Tue, Jun 20, 2017 at 10:43 AM, Craig Ringer <craig@2ndquadrant.com> wrote:
> Especially with a 6-week-old baby now....
Congratulations!
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-06-20 15:51 ` Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2017-06-20 15:51 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Craig Ringer <craig@2ndquadrant.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Andres Freund wrote:
> FWIW, I still think this needs a pgbench or similar example integration,
> so we can actually properly measure the benefits.
Here's an updated version of the patch I made during review,
adding \beginbatch and \endbatch to pgbench.
The performance improvement appears clearly
with a custom script of this kind:
\beginbatch
UPDATE pgbench_branches SET bbalance = bbalance + 1 WHERE bid = 0;
..above repeated 1000 times...
\endbatch
versus the same with a BEGIN; END; pair instead of \beginbatch \endbatch
On localhost on my desktop I tend to see a 30% difference in favor
of the batch mode with that kind of test.
On slower networks there are much bigger differences.
The latest main patch (v10) must also be slightly updated for HEAD,
because of this:
error: patch failed: src/interfaces/libpq/exports.txt:171
v11 attached without any other change.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments: download all (2 files, .tar.gz)
0001-Pipelining-batch-support-for-libpq-code-v11.patch (text/plain, 51.0K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index c3bd4f9..366f278 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4656,6 +4656,526 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSyncQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSyncQueue"><function>PQbatchSyncQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSyncQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSyncQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSyncQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSyncQueue">
+ <term>
+ <function>PQbatchSyncQueue</function>
+ <indexterm>
+ <primary>PQbatchSyncQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSyncQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchProcessQueue</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4696,6 +5216,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index f6fa0e4..d4ee576 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -936,6 +936,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0..49871f5 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,9 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQbatchQueueCount 173
+PQenterBatchMode 174
+PQexitBatchMode 175
+PQbatchSyncQueue 176
+PQbatchProcessQueue 177
+PQbatchStatus 178
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 02ec8f0..a92f4bc 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3483,6 +3483,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3494,6 +3513,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3526,6 +3546,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9decd53..2a3928b 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,302 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSyncQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return false;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2222,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2427,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2628,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2646,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2668,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2693,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2569,6 +3095,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index e7496c5..321fc60 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSyncQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 335568b..619f5c0 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -384,6 +406,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -684,6 +717,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
pgbench-batch-mode-v3.patch (text/plain, 5.5K) download
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 14aa587..016455e 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -269,7 +269,8 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
+ * CSTATE_WAIT_RESULT state unless in batch mode.
+ * On a \sleep meta-command, the timer is set,
* and we enter the CSTATE_SLEEP state to wait for it to expire. Other
* meta-commands are executed immediately.
*
@@ -1882,11 +1883,24 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- fprintf(stderr, "%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In batch mode, we use asynchronous functions. If a server-side
+ * error occurs, it will be processed later among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2165,7 +2179,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
return;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in batch mode */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -2207,7 +2227,51 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
else
{
- if (pg_strcasecmp(argv[0], "set") == 0)
+ if (pg_strcasecmp(argv[0], "beginbatch") == 0)
+ {
+ /*
+ * In batch mode, we use a workflow based on libpq batch
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "cannot use batch mode with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "already in batch mode");
+ break;
+ }
+ if (PQenterBatchMode(st->con) == 0)
+ {
+ commandFailed(st, "failed to start a batch");
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "endbatch") == 0)
+ {
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ commandFailed(st, "not in batch mode");
+ break;
+ }
+ if (!PQbatchSyncQueue(st->con))
+ {
+ commandFailed(st, "failed to end the batch");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQexitBatchMode(st->con) == 0)
+ {
+ /* collect pending results before getting out of batch mode */
+ st->state = CSTATE_WAIT_RESULT;
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "set") == 0)
{
PgBenchExpr *expr = command->expr;
PgBenchValue result;
@@ -2279,6 +2343,7 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
break;
+
/*
* Wait for the current SQL command to complete
*/
@@ -2295,6 +2360,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_ON &&
+ !PQbatchProcessQueue(st->con))
+ {
+ /* no complete result yet in batch mode*/
+ return;
+ }
+
/*
* Read and discard the query result;
*/
@@ -2307,7 +2379,22 @@ doCustom(TState *thread, CState *st, StatsData *agg)
/* OK */
PQclear(res);
discard_response(st);
- st->state = CSTATE_END_COMMAND;
+ /*
+ * In non-batch mode, only one result per command is expected.
+ * In batch mode, keep waiting for results until getting
+ * PGRES_BATCH_END.
+ */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_END_COMMAND;
+ break;
+ case PGRES_BATCH_END:
+ if (PQexitBatchMode(st->con) == 1)
+ {
+ /* all results collected, exit out of command and batch mode */
+ st->state = CSTATE_END_COMMAND;
+ }
+ else
+ fprintf(stderr, "client %d to exit batch mode", st->id);
break;
default:
commandFailed(st, PQerrorMessage(st->con));
@@ -3173,6 +3260,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "beginbatch") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "endbatch") == 0 )
+ {
+ if (my_command->argc > 1)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "unexpected argument", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2017-06-21 23:40 ` Andres Freund <andres@anarazel.de>
2017-06-21 23:44 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 182+ messages in thread
From: Andres Freund @ 2017-06-21 23:40 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Craig Ringer <craig@2ndquadrant.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 2017-06-20 17:51:23 +0200, Daniel Verite wrote:
> Andres Freund wrote:
>
> > FWIW, I still think this needs a pgbench or similar example integration,
> > so we can actually properly measure the benefits.
>
> Here's an updated version of the patch I made during review,
> adding \beginbatch and \endbatch to pgbench.
> The performance improvement appears clearly
> with a custom script of this kind:
> \beginbatch
> UPDATE pgbench_branches SET bbalance = bbalance + 1 WHERE bid = 0;
> ..above repeated 1000 times...
> \endbatch
>
> versus the same with a BEGIN; END; pair instead of \beginbatch \endbatch
>
> On localhost on my desktop I tend to see a 30% difference in favor
> of the batch mode with that kind of test.
> On slower networks there are much bigger differences.
This is seriously impressive. Just using the normal pgbench mixed
workload, wrapping a whole transaction into a batch *doubles* the
throughput. And that's locally over a unix socket - the gain over
actual network will be larger.
\set nbranches 1 * :scale
\set ntellers 10 * :scale
\set naccounts 100000 * :scale
\set aid random(1, :naccounts)
\set bid random(1, :nbranches)
\set tid random(1, :ntellers)
\set delta random(-5000, 5000)
\beginbatch
BEGIN;
UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
END;
\endbatch
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-06-21 23:44 ` Craig Ringer <craig@2ndquadrant.com>
1 sibling, 0 replies; 182+ messages in thread
From: Craig Ringer @ 2017-06-21 23:44 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Iwata, Aya <iwata.aya@jp.fujitsu.com>; pgsql-hackers; David Steele <david@pgmasters.net>; Daniel Verite <daniel@manitou-mail.org>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Haribabu Kommi <kommi.haribabu@gmail.com>; Dmitry Igrishin <dmitigr@gmail.com>; Manuel Kniep <m.kniep@web.de>; Michael Paquier <michael.paquier@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
On 22 Jun. 2017 07:40, "Andres Freund" <andres@anarazel.de> wrote:
On 2017-06-20 17:51:23 +0200, Daniel Verite wrote:
> Andres Freund wrote:
>
> > FWIW, I still think this needs a pgbench or similar example integration,
> > so we can actually properly measure the benefits.
>
> Here's an updated version of the patch I made during review,
> adding \beginbatch and \endbatch to pgbench.
> The performance improvement appears clearly
> with a custom script of this kind:
> \beginbatch
> UPDATE pgbench_branches SET bbalance = bbalance + 1 WHERE bid = 0;
> ..above repeated 1000 times...
> \endbatch
>
> versus the same with a BEGIN; END; pair instead of \beginbatch \endbatch
>
> On localhost on my desktop I tend to see a 30% difference in favor
> of the batch mode with that kind of test.
> On slower networks there are much bigger differences.
This is seriously impressive. Just using the normal pgbench mixed
workload, wrapping a whole transaction into a batch *doubles* the
throughput. And that's locally over a unix socket - the gain over
actual network will be larger.
In my original tests I got over a 300x improvement on WAN :) . I should
check if the same applies with pgbench.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-06-22 00:29 ` Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
1 sibling, 1 reply; 182+ messages in thread
From: Andres Freund @ 2017-06-22 00:29 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Craig Ringer <craig@2ndquadrant.com>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 2017-06-21 16:40:48 -0700, Andres Freund wrote:
> On 2017-06-20 17:51:23 +0200, Daniel Verite wrote:
> > Andres Freund wrote:
> >
> > > FWIW, I still think this needs a pgbench or similar example integration,
> > > so we can actually properly measure the benefits.
> >
> > Here's an updated version of the patch I made during review,
> > adding \beginbatch and \endbatch to pgbench.
> > The performance improvement appears clearly
> > with a custom script of this kind:
> > \beginbatch
> > UPDATE pgbench_branches SET bbalance = bbalance + 1 WHERE bid = 0;
> > ..above repeated 1000 times...
> > \endbatch
> >
> > versus the same with a BEGIN; END; pair instead of \beginbatch \endbatch
> >
> > On localhost on my desktop I tend to see a 30% difference in favor
> > of the batch mode with that kind of test.
> > On slower networks there are much bigger differences.
>
> This is seriously impressive. Just using the normal pgbench mixed
> workload, wrapping a whole transaction into a batch *doubles* the
> throughput. And that's locally over a unix socket - the gain over
> actual network will be larger.
I've not analyzed this further, but something with the way network is
done isn't yet quite right either in the pgbench patch or in the libpq
patch. You'll currently get IO like:
sendto(3, "B\0\0\0\22\0P1_2\0\0\0\0\0\0\1\0\0D\0\0\0\6P\0E\0\0\0\t\0"..., 36, MSG_NOSIGNAL, NULL, 0) = 36
sendto(3, "B\0\0\0\35\0P1_4\0\0\0\0\1\0\0\0\0073705952\0\1\0\0D\0"..., 47, MSG_NOSIGNAL, NULL, 0) = 47
sendto(3, "B\0\0\0\35\0P1_6\0\0\0\0\1\0\0\0\0077740854\0\1\0\0D\0"..., 47, MSG_NOSIGNAL, NULL, 0) = 47
sendto(3, "B\0\0\0\35\0P1_8\0\0\0\0\1\0\0\0\0071570280\0\1\0\0D\0"..., 47, MSG_NOSIGNAL, NULL, 0) = 47
sendto(3, "B\0\0\0\36\0P1_10\0\0\0\0\1\0\0\0\0072634305\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\36\0P1_12\0\0\0\0\1\0\0\0\0078960656\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\36\0P1_14\0\0\0\0\1\0\0\0\0073030370\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\35\0P1_16\0\0\0\0\1\0\0\0\006376125\0\1\0\0D\0"..., 47, MSG_NOSIGNAL, NULL, 0) = 47
sendto(3, "B\0\0\0\36\0P1_18\0\0\0\0\1\0\0\0\0072982423\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\36\0P1_20\0\0\0\0\1\0\0\0\0073860195\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\36\0P1_22\0\0\0\0\1\0\0\0\0072794433\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\36\0P1_24\0\0\0\0\1\0\0\0\0075475271\0\1\0\0D"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
sendto(3, "B\0\0\0\23\0P1_25\0\0\0\0\0\0\1\0\0D\0\0\0\6P\0E\0\0\0\t"..., 37, MSG_NOSIGNAL, NULL, 0) = 37
sendto(3, "S\0\0\0\4", 5, MSG_NOSIGNAL, NULL, 0) = 5
recvfrom(3, "2\0\0\0\4n\0\0\0\4C\0\0\0\nBEGIN\0002\0\0\0\4T\0\0\0!\0"..., 16384, 0, NULL, NULL) = 775
recvfrom(3, 0x559a02667ff2, 15630, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667fb1, 15695, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667f6c, 15764, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667f2b, 15829, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667eea, 15894, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667ea9, 15959, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667e68, 16024, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667e24, 16092, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667de3, 16157, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667da2, 16222, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667d5d, 16291, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667d1c, 16356, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(3, 0x559a02667d06, 16378, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
I.e. we're doing tiny write send() syscalls (they should be coalesced)
and then completely unnecessarily call recv() over and over again
without polling. To me it looks very much like we're just doing either
exactly once per command...
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-06-22 01:03 ` Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:23 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
0 siblings, 2 replies; 182+ messages in thread
From: Craig Ringer @ 2017-06-22 01:03 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 22 June 2017 at 08:29, Andres Freund <andres@anarazel.de> wrote:
> I.e. we're doing tiny write send() syscalls (they should be coalesced)
That's likely worth doing, but can probably wait for a separate patch.
The kernel will usually do some packet aggregation unless we use
TCP_NODELAY (which we don't and shouldn't), and the syscall overhead
is IMO not worth worrying about just yet.
> and then completely unnecessarily call recv() over and over again
> without polling. To me it looks very much like we're just doing either
> exactly once per command...
Yeah, that looks suspect.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2017-06-22 01:07 ` Andres Freund <andres@anarazel.de>
2017-06-22 01:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
1 sibling, 2 replies; 182+ messages in thread
From: Andres Freund @ 2017-06-22 01:07 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 2017-06-22 09:03:05 +0800, Craig Ringer wrote:
> On 22 June 2017 at 08:29, Andres Freund <andres@anarazel.de> wrote:
>
> > I.e. we're doing tiny write send() syscalls (they should be coalesced)
>
> That's likely worth doing, but can probably wait for a separate patch.
I don't think so, we should get this right, it could have API influence.
> The kernel will usually do some packet aggregation unless we use
> TCP_NODELAY (which we don't and shouldn't), and the syscall overhead
> is IMO not worth worrying about just yet.
1)
/*
* Select socket options: no delay of outgoing data for
* TCP sockets, nonblock mode, close-on-exec. Fail if any
* of this fails.
*/
if (!IS_AF_UNIX(addr_cur->ai_family))
{
if (!connectNoDelay(conn))
{
pqDropConnection(conn, true);
conn->addr_cur = addr_cur->ai_next;
continue;
}
}
2) Even if nodelay weren't set, this can still lead to smaller packets
being sent, because you start sending normal sized tcp packets,
rather than jumbo ones, even if configured (pretty common these
days).
3) Syscall overhead is actually quite significant.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-06-22 01:18 ` Craig Ringer <craig@2ndquadrant.com>
1 sibling, 0 replies; 182+ messages in thread
From: Craig Ringer @ 2017-06-22 01:18 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 22 June 2017 at 09:07, Andres Freund <andres@anarazel.de> wrote:
> On 2017-06-22 09:03:05 +0800, Craig Ringer wrote:
>> On 22 June 2017 at 08:29, Andres Freund <andres@anarazel.de> wrote:
>>
>> > I.e. we're doing tiny write send() syscalls (they should be coalesced)
>>
>> That's likely worth doing, but can probably wait for a separate patch.
>
> I don't think so, we should get this right, it could have API influence.
>
>
>> The kernel will usually do some packet aggregation unless we use
>> TCP_NODELAY (which we don't and shouldn't), and the syscall overhead
>> is IMO not worth worrying about just yet.
>
> 1)
> /*
> * Select socket options: no delay of outgoing data for
> * TCP sockets, nonblock mode, close-on-exec. Fail if any
> * of this fails.
> */
> if (!IS_AF_UNIX(addr_cur->ai_family))
> {
> if (!connectNoDelay(conn))
> {
> pqDropConnection(conn, true);
> conn->addr_cur = addr_cur->ai_next;
> continue;
> }
> }
>
> 2) Even if nodelay weren't set, this can still lead to smaller packets
> being sent, because you start sending normal sized tcp packets,
> rather than jumbo ones, even if configured (pretty common these
> days).
>
> 3) Syscall overhead is actually quite significant.
Fair enough, and *headdesk* re not checking NODELAY. I thought I'd
checked for our use of that before, but I must've remembered wrong.
We could use TCP_CORK but it's not portable and it'd be better to just
collect up a buffer to dispatch.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-06-22 01:21 ` Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
1 sibling, 1 reply; 182+ messages in thread
From: Andres Freund @ 2017-06-22 01:21 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 2017-06-21 18:07:21 -0700, Andres Freund wrote:
> On 2017-06-22 09:03:05 +0800, Craig Ringer wrote:
> > On 22 June 2017 at 08:29, Andres Freund <andres@anarazel.de> wrote:
> >
> > > I.e. we're doing tiny write send() syscalls (they should be coalesced)
> >
> > That's likely worth doing, but can probably wait for a separate patch.
>
> I don't think so, we should get this right, it could have API influence.
>
>
> > The kernel will usually do some packet aggregation unless we use
> > TCP_NODELAY (which we don't and shouldn't), and the syscall overhead
> > is IMO not worth worrying about just yet.
>
> 1)
> /*
> * Select socket options: no delay of outgoing data for
> * TCP sockets, nonblock mode, close-on-exec. Fail if any
> * of this fails.
> */
> if (!IS_AF_UNIX(addr_cur->ai_family))
> {
> if (!connectNoDelay(conn))
> {
> pqDropConnection(conn, true);
> conn->addr_cur = addr_cur->ai_next;
> continue;
> }
> }
>
> 2) Even if nodelay weren't set, this can still lead to smaller packets
> being sent, because you start sending normal sized tcp packets,
> rather than jumbo ones, even if configured (pretty common these
> days).
>
> 3) Syscall overhead is actually quite significant.
Proof of the pudding:
pgbench of 10 pgbench select statements in a batch:
as submitted by Daniel:
pgbench -h localhost -M prepared -S -n -c 16 -j 16 -T 10000 -P 1 -f ~/tmp/pgbench-select-only-batch.sq
progress: 1.0 s, 24175.5 tps, lat 0.647 ms stddev 0.782
progress: 2.0 s, 27737.6 tps, lat 0.577 ms stddev 0.625
progress: 3.0 s, 28853.3 tps, lat 0.554 ms stddev 0.619
progress: 4.0 s, 26660.8 tps, lat 0.600 ms stddev 0.776
progress: 5.0 s, 30023.8 tps, lat 0.533 ms stddev 0.484
progress: 6.0 s, 29959.3 tps, lat 0.534 ms stddev 0.450
progress: 7.0 s, 29944.9 tps, lat 0.534 ms stddev 0.536
progress: 8.0 s, 30137.7 tps, lat 0.531 ms stddev 0.533
progress: 9.0 s, 30285.2 tps, lat 0.528 ms stddev 0.479
progress: 10.0 s, 30228.7 tps, lat 0.529 ms stddev 0.460
progress: 11.0 s, 29921.4 tps, lat 0.534 ms stddev 0.613
progress: 12.0 s, 29982.4 tps, lat 0.533 ms stddev 0.510
progress: 13.0 s, 29247.4 tps, lat 0.547 ms stddev 0.526
progress: 14.0 s, 28757.3 tps, lat 0.556 ms stddev 0.635
progress: 15.0 s, 29035.3 tps, lat 0.551 ms stddev 0.523
^C
sample vmstat:
r b swpd free buff cache si so bi bo in cs us sy id wa st
19 0 0 488992 787332 23558676 0 0 0 0 9720 455099 65 35 0 0 0
(i.e. ~450k context switches)
hackily patched:
pgbench -h localhost -M prepared -S -n -c 16 -j 16 -T 10000 -P 1 -f ~/tmp/pgbench-select-only-batch.sq
progress: 1.0 s, 40545.2 tps, lat 0.386 ms stddev 0.625
progress: 2.0 s, 48158.0 tps, lat 0.332 ms stddev 0.277
progress: 3.0 s, 50125.7 tps, lat 0.319 ms stddev 0.204
progress: 4.0 s, 50740.6 tps, lat 0.315 ms stddev 0.250
progress: 5.0 s, 50795.6 tps, lat 0.315 ms stddev 0.246
progress: 6.0 s, 51195.6 tps, lat 0.312 ms stddev 0.207
progress: 7.0 s, 50746.7 tps, lat 0.315 ms stddev 0.264
progress: 8.0 s, 50619.1 tps, lat 0.316 ms stddev 0.250
progress: 9.0 s, 50619.4 tps, lat 0.316 ms stddev 0.228
progress: 10.0 s, 46967.8 tps, lat 0.340 ms stddev 0.499
progress: 11.0 s, 50480.1 tps, lat 0.317 ms stddev 0.239
progress: 12.0 s, 50242.5 tps, lat 0.318 ms stddev 0.286
progress: 13.0 s, 49912.7 tps, lat 0.320 ms stddev 0.266
progress: 14.0 s, 49841.7 tps, lat 0.321 ms stddev 0.271
progress: 15.0 s, 49807.1 tps, lat 0.321 ms stddev 0.248
^C
sample vmstat:
r b swpd free buff cache si so bi bo in cs us sy id wa st
23 0 0 482008 787312 23558996 0 0 0 0 8219 105097 87 14 0 0 0
(i.e. ~100k context switches)
That's *localhost*.
It's completely possible that I've screwed something up here, I didn't
test it besides running pgbench, but the send/recv'd data looks like
it's similar amounts of data, just fewer syscalls.
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
fewer-syscalls.diff (text/x-diff, 1.8K) download
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e498ad61e5..aeed1649ce 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -2352,14 +2352,17 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (debug)
fprintf(stderr, "client %d receiving\n", st->id);
- if (!PQconsumeInput(st->con))
- { /* there's something wrong */
- commandFailed(st, "perhaps the backend died while processing");
- st->state = CSTATE_ABORTED;
- break;
- }
if (PQisBusy(st->con))
- return; /* don't have the whole result yet */
+ {
+ if (!PQconsumeInput(st->con))
+ { /* there's something wrong */
+ commandFailed(st, "perhaps the backend died while processing");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQisBusy(st->con))
+ return; /* don't have the whole result yet */
+ }
if (PQbatchStatus(st->con) == PQBATCH_MODE_ON &&
!PQbatchProcessQueue(st->con))
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 4cb87a4393..210410a92c 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -1725,12 +1725,15 @@ PQsendQueryGuts(PGconn *conn,
else
*last_query = NULL;
- /*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
- */
- if (pqFlush(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ }
/* OK, it's launched! */
if (conn->batch_status != PQBATCH_MODE_OFF)
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-06-22 11:43 ` Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2017-06-22 11:43 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Andres Freund wrote:
- if (pqFlush(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /*
+ * Give the data a push. In nonblock mode, don't complain if
we're unable
+ * to send it all; PQgetResult() will do any additional
flushing needed.
+ */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ }
Seems to be responsible for roughly an 1.7x speedup in tps and equivalent
decrease in latency, based on the "progress" info.
I wonder how much of that corresponds to a decrease in the number of
packets versus the number of syscalls. Both matter, I guess.
But OTOH there are certainly batch workloads where it will be preferrable
for the first query to reach the server ASAP, rather than waiting to be
coalesced with the next ones.
libpq is not going to know what's best.
One option may be to leave that decision to the user by providing a
PQBatchAutoFlush(true|false) property, along with a PQBatchFlush()
function. Maybe we could even let the user set the size of the sending
buffer, so those who really want to squeeze performance may tune it
for their network and workload.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2017-06-22 15:52 ` Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
0 siblings, 1 reply; 182+ messages in thread
From: Andres Freund @ 2017-06-22 15:52 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Hi,
On 2017-06-22 13:43:35 +0200, Daniel Verite wrote:
> But OTOH there are certainly batch workloads where it will be preferrable
> for the first query to reach the server ASAP, rather than waiting to be
> coalesced with the next ones.
Is that really something people expect from a batch API? I suspect it's
not really, and nothing would stop one from adding PQflush() or similar
calls if desirable anyway.
FWIW, the way I did that in the hack clearly isn't ok: If you were to
send a gigabyte of queries, it'd buffer them all up in memory... So some
more intelligence is going to be needed.
> libpq is not going to know what's best.
> One option may be to leave that decision to the user by providing a
> PQBatchAutoFlush(true|false) property, along with a PQBatchFlush()
> function.
What'd be the difference between PQflush() and PQbatchFlush()?
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-06-22 17:42 ` Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2017-06-22 17:42 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Andres Freund wrote:
> > One option may be to leave that decision to the user by providing a
> > PQBatchAutoFlush(true|false) property, along with a PQBatchFlush()
> > function.
>
> What'd be the difference between PQflush() and PQbatchFlush()?
I guess no difference, I was just not seeing that libpq already provides
this functionality...
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2017-08-10 05:23 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-08-10 05:23 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Andres Freund <andres@anarazel.de>; Craig Ringer <craig@2ndquadrant.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Andres Freund wrote :
> If you were to send a gigabyte of queries, it'd buffer them all up in
memory... So some
>more intelligence is going to be needed.
Firstly, sorry for the delayed response as I got busy with other
activities.
To buffer up the queries before flushing them to the socket, I think
"conn->outCount>=65536" is ok to use, as 64k is considered to be safe in
Windows as per comments in pqSendSome() API.
Attached the code patch to replace pqFlush calls with pqBatchFlush in the
asynchronous libpq function calls flow.
Still pqFlush is used in "PQbatchSyncQueue" and
"PQexitBatchMode" functions.
Am failing to see the benefit in allowing user to set
PQBatchAutoFlush(true|false) property? Is it really needed?
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v12.patch (application/octet-stream, 52.9K) download
From ea300052d23b7ee311b83a9a8fa3cc63e04e0ae6 Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Thu, 10 Aug 2017 10:10:51 +1000
Subject: [PATCH] Pipelining-batch-support-for-libpq-code-v12
---
doc/src/sgml/libpq.sgml | 528 +++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 624 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 +-
10 files changed, 1236 insertions(+), 43 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index ad5e9b9..435df5b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4723,6 +4723,526 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSyncQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSyncQueue"><function>PQbatchSyncQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSyncQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSyncQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSyncQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSyncQueue">
+ <term>
+ <function>PQbatchSyncQueue</function>
+ <indexterm>
+ <primary>PQbatchSyncQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSyncQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchProcessQueue</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4763,6 +5283,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index de03362..ba45ec2 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -939,6 +939,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0..49871f5 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,9 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQbatchQueueCount 173
+PQenterBatchMode 174
+PQexitBatchMode 175
+PQbatchSyncQueue 176
+PQbatchProcessQueue 177
+PQbatchStatus 178
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 5a964bf..780ab35 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3566,6 +3566,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3577,6 +3596,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3608,6 +3628,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index e1e2d18..db9b7c9 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,7 +71,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,31 +1303,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,59 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1562,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1575,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,35 +1704,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1866,309 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return true;
+
+sendFailed:
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSyncQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return false;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2228,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2433,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2634,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2652,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2674,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2167,14 +2695,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2569,6 +3101,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3762,3 +4301,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount>=65536))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index a58f701..ffd6737 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index a484fe8..6d90ae7 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 1d915e7..7eb465a 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSyncQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 4291360..425e318 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -385,6 +407,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -683,6 +716,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-08-23 09:40 ` Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Andres Freund @ 2017-08-23 09:40 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Craig Ringer <craig@2ndquadrant.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Hi,
On 2017-08-10 15:23:06 +1000, Vaishnavi Prabakaran wrote:
> Andres Freund wrote :
> > If you were to send a gigabyte of queries, it'd buffer them all up in
> memory... So some
> >more intelligence is going to be needed.
>
> Firstly, sorry for the delayed response as I got busy with other
> activities.
No worries - development of new features was slowed down anyway, due to
the v10 feature freeze.
> To buffer up the queries before flushing them to the socket, I think
> "conn->outCount>=65536" is ok to use, as 64k is considered to be safe in
> Windows as per comments in pqSendSome() API.
>
> Attached the code patch to replace pqFlush calls with pqBatchFlush in the
> asynchronous libpq function calls flow.
> Still pqFlush is used in "PQbatchSyncQueue" and
> "PQexitBatchMode" functions.
> Am failing to see the benefit in allowing user to set
> PQBatchAutoFlush(true|false) property? Is it really needed?
I'm inclined not to introduce that for now. If somebody comes up with a
convincing usecase and numbers, we can add it later. Libpq API is set in
stone, so I'd rather not introduce unnecessary stuff...
> + <para>
> + Much like asynchronous query mode, there is no performance disadvantage to
> + using batching and pipelining. It increases client application complexity
> + and extra caution is required to prevent client/server deadlocks but
> + can sometimes offer considerable performance improvements.
> + </para>
That's not necessarily true, is it? Unless you count always doing
batches of exactly size 1.
> + <para>
> + Batching is most useful when the server is distant, i.e. network latency
> + (<quote>ping time</quote>) is high, and when many small operations are being performed in
> + rapid sequence. There is usually less benefit in using batches when each
> + query takes many multiples of the client/server round-trip time to execute.
> + A 100-statement operation run on a server 300ms round-trip-time away would take
> + 30 seconds in network latency alone without batching; with batching it may spend
> + as little as 0.3s waiting for results from the server.
> + </para>
I'd add a remark that this is frequently beneficial even in cases of
minimal latency - as e.g. shown by the numbers I presented upthread.
> + <para>
> + Use batches when your application does lots of small
> + <literal>INSERT</literal>, <literal>UPDATE</literal> and
> + <literal>DELETE</literal> operations that can't easily be transformed into
> + operations on sets or into a
> + <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
> + </para>
Aren't SELECTs also a major beneficiarry of this?
> + <para>
> + Batching is less useful when information from one operation is required by the
> + client before it knows enough to send the next operation.
s/less/not/
> + <note>
> + <para>
> + The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
> + use batches on server versions 8.4 and newer. Batching works on any server
> + that supports the v3 extended query protocol.
> + </para>
> + </note>
Where's the 8.4 coming from?
> + <para>
> + The client uses libpq's asynchronous query functions to dispatch work,
> + marking the end of each batch with <function>PQbatchSyncQueue</function>.
> + And to get results, it uses <function>PQgetResult</function> and
> + <function>PQbatchProcessQueue</function>. It may eventually exit
> + batch mode with <function>PQexitBatchMode</function> once all results are
> + processed.
> + </para>
> +
> + <note>
> + <para>
> + It is best to use batch mode with <application>libpq</application> in
> + <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
> + blocking mode it is possible for a client/server deadlock to occur. The
> + client will block trying to send queries to the server, but the server will
> + block trying to send results from queries it has already processed to the
> + client. This only occurs when the client sends enough queries to fill its
> + output buffer and the server's receive buffer before switching to
> + processing input from the server, but it's hard to predict exactly when
> + that'll happen so it's best to always use non-blocking mode.
> + </para>
> + </note>
Mention that nonblocking only actually helps if send/recv is done as
required, and can essentially require unbound memory? We probably
should either document or implement some smarts about when to signal
read/write readyness. Otherwise we e.g. might be receiving tons of
result data without having sent the next query - or the other way round.
> + <sect3 id="libpq-batch-sending">
> + <title>Issuing queries</title>
> +
> + <para>
> + After entering batch mode the application dispatches requests
> + using normal asynchronous <application>libpq</application> functions such as
> + <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
> + <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
> + <function>PQsendDescribePrepared</function>.
> + The asynchronous requests are followed by a <link
> + linkend="libpq-PQbatchSyncQueue"><function>PQbatchSyncQueue(conn)</function></link> call to mark
> + the end of the batch. The client <emphasis>does not</emphasis> need to call
> + <function>PQgetResult</function> immediately after dispatching each
> + operation. <link linkend="libpq-batch-results">Result processing</link>
> + is handled separately.
> + </para>
> +
> + <para>
> + Batched operations will be executed by the server in the order the client
> + sends them. The server will send the results in the order the statements
> + executed. The server may begin executing the batch before all commands
> + in the batch are queued and the end of batch command is sent. If any
> + statement encounters an error the server aborts the current transaction and
> + skips processing the rest of the batch. Query processing resumes after the
> + end of the failed batch.
> + </para>
Maybe note that multiple batches can be "in flight"?
I.e. PQbatchSyncQueue() is about error handling, nothing else? Don't
have a great idea, but we might want to rename...
> + <note>
> + <para>
> + The client must not assume that work is committed when it
> + <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
> + corresponding result is received to confirm the commit is complete.
> + Because errors arrive asynchronously the application needs to be able to
> + restart from the last <emphasis>received</emphasis> committed change and
> + resend work done after that point if something goes wrong.
> + </para>
> + </note>
This seems fairly independent of batching.
> + </sect3>
> +
> + <sect3 id="libpq-batch-interleave">
> + <title>Interleaving result processing and query dispatch</title>
> +
> + <para>
> + To avoid deadlocks on large batches the client should be structured around
> + a nonblocking I/O loop using a function like <function>select</function>,
> + <function>poll</function>, <function>epoll</function>,
> + <function>WaitForMultipleObjectEx</function>, etc.
> + </para>
> +
> + <para>
> + The client application should generally maintain a queue of work still to
> + be dispatched and a queue of work that has been dispatched but not yet had
> + its results processed.
Hm. Why? If queries are just issued, no such queue is required?
> When the socket is writable it should dispatch more
> + work. When the socket is readable it should read results and process them,
> + matching them up to the next entry in its expected results queue. Batches
> + should be scoped to logical units of work, usually (but not always) one
> + transaction per batch. There's no need to exit batch mode and re-enter it
> + between batches or to wait for one batch to finish before sending the next.
> + </para>
This really needs to take memory usage into account.
> + </variablelist>
> +
> + </listitem>
> + </varlistentry>
> +
> + <varlistentry id="libpq-PQenterBatchMode">
> + <term>
> + <function>PQenterBatchMode</function>
> + <indexterm>
> + <primary>PQenterBatchMode</primary>
> + </indexterm>
> + </term>
> +
> + <listitem>
> + <para>
> + Causes a connection to enter batch mode if it is currently idle or
> + already in batch mode.
> +
> +<synopsis>
> +int PQenterBatchMode(PGconn *conn);
> +</synopsis>
> +
> + </para>
> + <para>
> + Returns 1 for success. Returns 0 and has no
> + effect if the connection is not currently idle, i.e. it has a result
> + ready, is waiting for more input from the server, etc. This function
> + does not actually send anything to the server, it just changes the
> + <application>libpq</application> connection state.
> +
> + </para>
> + </listitem>
> + </varlistentry>
> +
> + <varlistentry id="libpq-PQexitBatchMode">
> + <term>
> + <function>PQexitBatchMode</function>
> + <indexterm>
> + <primary>PQexitBatchMode</primary>
> + </indexterm>
> + </term>
> +
> + <listitem>
> + <para>
> + Causes a connection to exit batch mode if it is currently in batch mode
> + with an empty queue and no pending results.
> +<synopsis>
> +int PQexitBatchMode(PGconn *conn);
> +</synopsis>
> + </para>
> + <para>Returns 1 for success.
> + Returns 1 and takes no action if not in batch mode. If the connection has
"returns 1"?
> + <varlistentry id="libpq-PQbatchSyncQueue">
> + <term>
> + <function>PQbatchSyncQueue</function>
> + <indexterm>
> + <primary>PQbatchSyncQueue</primary>
> + </indexterm>
> + </term>
> +
> + <listitem>
> + <para>
> + Delimits the end of a set of a batched commands by sending a <link
> + linkend="protocol-flow-ext-query">sync message</link> and flushing
> + the send buffer. The end of a batch serves as
> + the delimiter of an implicit transaction and
> + an error recovery point; see <link linkend="libpq-batch-errors">
> + error handling</link>.
I wonder why this isn't framed as PQbatchIssue/Send/...()? Syncing seems
to mostly make sense from a protocol POV.
> + <varlistentry id="libpq-PQbatchQueueCount">
> + <term>
> + <function>PQbatchQueueCount</function>
> + <indexterm>
> + <primary>PQbatchQueueCount</primary>
> + </indexterm>
> + </term>
> +
> + <listitem>
> + <para>
> + Returns the number of queries still in the queue for this batch, not
> + including any query that's currently having results being processed.
> + This is the number of times <function>PQbatchProcessQueue</function> has to be
> + called before the query queue is empty again.
> +
> +<synopsis>
> +int PQbatchQueueCount(PGconn *conn);
> +</synopsis>
> +
> + </para>
> + </listitem>
> + </varlistentry>
Given that apps are supposed to track this, I'm not sure why we have
this?
> +/*
> + * PQrecyclePipelinedCommand
> + * Push a command queue entry onto the freelist. It must be a dangling entry
> + * with null next pointer and not referenced by any other entry's next pointer.
> + */
> +static void
> +PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
> +{
> + if (entry == NULL)
> + return;
> + if (entry->next != NULL)
> + {
> + fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
> + abort();
Don't think we use abort() in libpq like that. There's some Assert()s
tho.
> static bool
> PQsendQueryStart(PGconn *conn)
> @@ -1377,20 +1486,59 @@ PQsendQueryStart(PGconn *conn)
> libpq_gettext("no connection to the server\n"));
> return false;
> }
> - /* Can't send while already busy, either. */
> - if (conn->asyncStatus != PGASYNC_IDLE)
> + /* Can't send while already busy, either, unless enqueuing for later */
> + if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
> {
> printfPQExpBuffer(&conn->errorMessage,
> libpq_gettext("another command is already in progress\n"));
> return false;
> }
>
> - /* initialize async result-accumulation state */
> - pqClearAsyncResult(conn);
> + if (conn->batch_status != PQBATCH_MODE_OFF)
> + {
> + /*
Weirdly indented.
> + if (conn->batch_status != PQBATCH_MODE_OFF)
> + {
> + pipeCmd = PQmakePipelinedCommand(conn);
> +
> + if (pipeCmd == NULL)
> + return 0; /* error msg already set */
> +
> + last_query = &pipeCmd->query;
> + queryclass = &pipeCmd->queryclass;
> + }
> + else
> + {
> + last_query = &conn->last_query;
> + queryclass = &conn->queryclass;
> + }
The amount of complexity / branches we're adding to all of these is more
than a bit unsightly.
> +/*
> + * PQbatchQueueCount
> + * Return number of queries currently pending in batch mode
> + */
> +int
> +PQbatchQueueCount(PGconn *conn)
> +{
> + int count = 0;
> + PGcommandQueueEntry *entry;
> +
> + if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
> + return 0;
> +
> + entry = conn->cmd_queue_head;
> + while (entry != NULL)
> + {
> + ++count;
> + entry = entry->next;
> + }
> + return count;
> +}
Ugh, O(N)? In that case I'd rather just remove this.
> +/*
> + * PQbatchBegin
Mismatched w/ actual function name.
> + * Put an idle connection in batch mode. Commands submitted after this
> + * can be pipelined on the connection, there's no requirement to wait for
> + * one to finish before the next is dispatched.
> + *
> + * Queuing of new query or syncing during COPY is not allowed.
+"a"?
> + * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
> + * commands may be sent while in batch mode. Batch mode can be exited by
> + * calling PQbatchEnd() once all results are processed.
> + *
> + * This doesn't actually send anything on the wire, it just puts libpq
> + * into a state where it can pipeline work.
> + */
> +int
> +PQenterBatchMode(PGconn *conn)
> +{
> + if (!conn)
> + return false;
true/false isn't quite in line with int return code.
> +/*
> + * PQbatchEnd
wrong name.
> + * End batch mode and return to normal command mode.
> + *
> + * Has no effect unless the client has processed all results
> + * from all outstanding batches and the connection is idle.
That seems wrong - will lead to hard to diagnose errors.
> + * Returns true if batch mode ended.
> + */
> +int
> +PQexitBatchMode(PGconn *conn)
> +{
> + if (!conn)
> + return false;
> +
> + if (conn->batch_status == PQBATCH_MODE_OFF)
> + return true;
> +
> + switch (conn->asyncStatus)
> + {
> + case PGASYNC_IDLE:
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext_noop("internal error, IDLE in batch mode"));
> + break;
> + case PGASYNC_COPY_IN:
> + case PGASYNC_COPY_OUT:
> + case PGASYNC_COPY_BOTH:
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext_noop("internal error, COPY in batch mode"));
> + break;
So we'll still check the queue in this case, that's a bit weird?
> +/*
> + * PQbatchQueueSync
Out of sync.
> + * End a batch submission by sending a protocol sync. The connection will
> + * remain in batch mode and unavailable for new non-batch commands until all
> + * results from the batch are processed by the client.
"unavailable for new non-batch commands" - that's hard to follow, and
seems pretty redundant with PQendBatchMode (or however it's called).
> + * It's legal to start submitting another batch immediately, without waiting
> + * for the results of the current batch. There's no need to end batch mode
> + * and start it again.
> + *
> + * If a command in a batch fails, every subsequent command up to and including
> + * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
> + * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
> + * produced.
Hm, should probably mention that that's only true for commands since the
last PQbatchQueueSync?
> +/*
> + * PQbatchQueueProcess
Out of sync.
> + * Returns false if the current query isn't done yet, the connection
> + * is not in a batch, or there are no more queries to process.
Last complaint about this - think this forgiving mode is a mistake.
> + */
> +int
> +PQbatchProcessQueue(PGconn *conn)
> +{
> + /* This command's results will come in immediately.
> + * Initialize async result-accumulation state */
> + pqClearAsyncResult(conn);
I'm not following?
> /*
> * PQgetResult
> @@ -1749,10 +2228,32 @@ PQgetResult(PGconn *conn)
> + if (conn->batch_status != PQBATCH_MODE_OFF)
> + {
> + /*
> + * batched queries aren't followed by a Sync to put us back in
> + * PGASYNC_IDLE state, and when we do get a sync we could
> + * still have another batch coming after this one.
This needs rephrasing.
> + * The connection isn't idle since we can't submit new
> + * nonbatched commands. It isn't also busy since the current
> + * command is done and we need to process a new one.
> + */
> + conn->asyncStatus = PGASYNC_QUEUED;
Not sure I like the name.
> + if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
> + {
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
> + return false;
> + }
Why do we need the PGASYNC_QUEUED test here?
> +/* pqBatchFlush
> + * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
> + * In non-batch mode, data will be flushed all the time.
> + */
> +static int
> +pqBatchFlush(PGconn *conn)
> +{
> + if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount>=65536))
> + return(pqFlush(conn));
> + return 0; /* Just to keep compiler quiet */
> +}
This should be defined in a macro or such, rather than hardcoded.
Falling over now. This seems like enough feedback for a bit of work
anyway.
Regards,
Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2017-09-13 05:06 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-09-13 05:06 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Daniel Verite <daniel@manitou-mail.org>; Craig Ringer <craig@2ndquadrant.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Wed, Aug 23, 2017 at 7:40 PM, Andres Freund <andres@anarazel.de> wrote:
>
>
>
> > Am failing to see the benefit in allowing user to set
> > PQBatchAutoFlush(true|false) property? Is it really needed?
>
> I'm inclined not to introduce that for now. If somebody comes up with a
> convincing usecase and numbers, we can add it later. Libpq API is set in
> stone, so I'd rather not introduce unnecessary stuff...
>
>
Thanks for reviewing the patch and yes ok.
>
>
> > + <para>
> > + Much like asynchronous query mode, there is no performance
> disadvantage to
> > + using batching and pipelining. It increases client application
> complexity
> > + and extra caution is required to prevent client/server deadlocks but
> > + can sometimes offer considerable performance improvements.
> > + </para>
>
> That's not necessarily true, is it? Unless you count always doing
> batches of exactly size 1.
>
Client application complexity is increased in batch mode,because
application needs to remember the query queue status. Results processing
can be done at anytime, so the application needs to know till what query,
the results are consumed.
> + <para>
> > + Use batches when your application does lots of small
> > + <literal>INSERT</literal>, <literal>UPDATE</literal> and
> > + <literal>DELETE</literal> operations that can't easily be
> transformed into
> > + operations on sets or into a
> > + <link linkend="libpq-copy"><literal>COPY</literal></link>
> operation.
> > + </para>
>
> Aren't SELECTs also a major beneficiarry of this?
>
Hmm, though SELECTs also benefit from batch mode, doing multiple selects in
batch mode will fill up the memory rapidly and might not be as beneficial
as other operations listed.
> > + <para>
> > + Batching is less useful when information from one operation is
> required by the
> > + client before it knows enough to send the next operation.
>
> s/less/not/
>
>
Corrected.
>
> > + <note>
> > + <para>
> > + The batch API was introduced in PostgreSQL 10.0, but clients using
> PostgresSQL 10.0 version of libpq can
> > + use batches on server versions 8.4 and newer. Batching works on
> any server
> > + that supports the v3 extended query protocol.
> > + </para>
> > + </note>
>
> Where's the 8.4 coming from?
>
>
I guess it is 7.4 where "PQsendQueryParams" is introduced, and not 8.4.
Corrected.
> + <note>
> > + <para>
> > + It is best to use batch mode with <application>libpq</application>
> in
> > + <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>.
> If used in
> > + blocking mode it is possible for a client/server deadlock to
> occur. The
> > + client will block trying to send queries to the server, but the
> server will
> > + block trying to send results from queries it has already processed
> to the
> > + client. This only occurs when the client sends enough queries to
> fill its
> > + output buffer and the server's receive buffer before switching to
> > + processing input from the server, but it's hard to predict exactly
> when
> > + that'll happen so it's best to always use non-blocking mode.
> > + </para>
> > + </note>
>
> Mention that nonblocking only actually helps if send/recv is done as
> required, and can essentially require unbound memory? We probably
> should either document or implement some smarts about when to signal
> read/write readyness. Otherwise we e.g. might be receiving tons of
> result data without having sent the next query - or the other way round.
>
>
Added a statement for caution in documentation and again this is one of the
reason why SELECT query is not so beneficial in batch mode.
> Maybe note that multiple batches can be "in flight"?
> I.e. PQbatchSyncQueue() is about error handling, nothing else? Don't
> have a great idea, but we might want to rename...
>
>
This function not only does error handling, but also sends the "Sync"
message to backend. In batch mode, "Sync" message is not sent with every
query but will
be sent only via this function to mark the end of implicit transaction.
Renamed it to PQbatchCommitQueue. Kindly let me know if you think of any
other better name.
> > + <note>
> > + <para>
> > + The client must not assume that work is committed when it
> > + <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when
> the
> > + corresponding result is received to confirm the commit is
> complete.
> > + Because errors arrive asynchronously the application needs to be
> able to
> > + restart from the last <emphasis>received</emphasis> committed
> change and
> > + resend work done after that point if something goes wrong.
> > + </para>
> > + </note>
>
> This seems fairly independent of batching.
>
>
Yes and the reason why is it explicitly specified for batch mode is that if
more than one explicit transactions are used in Single batch, then failure
of one transaction will lead to skipping the consequent transactions until
the end of current batch is reached. This behavior is specific to batch
mode, so adding a precautionary note here is needed I think.
> > + </sect3>
> > +
> > + <sect3 id="libpq-batch-interleave">
> > + <title>Interleaving result processing and query dispatch</title>
> > +
> > + <para>
> > + To avoid deadlocks on large batches the client should be
> structured around
> > + a nonblocking I/O loop using a function like
> <function>select</function>,
> > + <function>poll</function>, <function>epoll</function>,
> > + <function>WaitForMultipleObjectEx</function>, etc.
> > + </para>
> > +
> > + <para>
> > + The client application should generally maintain a queue of work
> still to
> > + be dispatched and a queue of work that has been dispatched but not
> yet had
> > + its results processed.
>
> Hm. Why? If queries are just issued, no such queue is required?
>
This is essentially telling that client application should remember the
order of queries sent , so that result processing will be based on what
results are expected. For e.g, if the order of queries are 1)SELECT,
2)INSERT, then in result processing, "PGRES_TUPLES_OK" and later
"PGRES_COMMAND_OK" checks are required. Above mentioned queue of work means
the intelligent to remember the queue of queries.
> > When the socket is writable it should dispatch more
> > + work. When the socket is readable it should read results and
> process them,
> > + matching them up to the next entry in its expected results queue.
> Batches
> > + should be scoped to logical units of work, usually (but not
> always) one
> > + transaction per batch. There's no need to exit batch mode and
> re-enter it
> > + between batches or to wait for one batch to finish before sending
> the next.
> > + </para>
>
> This really needs to take memory usage into account.
>
>
Modified the para to include that frequency of result processing should be
based on available memory.
> > +<synopsis>
> > +int PQenterBatchMode(PGconn *conn);
> > +</synopsis>
> > +<synopsis>
> > +int PQexitBatchMode(PGconn *conn);
> > +</synopsis>
> > + </para>
> > + <para>Returns 1 for success.
> > + Returns 1 and takes no action if not in batch mode. If the
> connection has
>
> "returns 1"?
>
Modified the code to return 1/0.
>
>
> > + <varlistentry id="libpq-PQbatchSyncQueue">
> > + <term>
> > + <function>PQbatchSyncQueue</function>
> > + <indexterm>
> > + <primary>PQbatchSyncQueue</primary>
> > + </indexterm>
> > + </term>
>
> I wonder why this isn't framed as PQbatchIssue/Send/...()? Syncing seems
> to mostly make sense from a protocol POV.
>
>
Renamed to PQbatchCommitQueue.
> > +<synopsis>
> > +int PQbatchQueueCount(PGconn *conn);
> > +</synopsis>
> > +
>
> Given that apps are supposed to track this, I'm not sure why we have
> this?
>
>
Removed this function considering your another comment about O(N) as well.
> > +static void
> > +PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
> > +{
> > + if (entry == NULL)
> > + return;
> > + if (entry->next != NULL)
> > + {
> > + fprintf(stderr, libpq_gettext("tried to recycle
> non-dangling command queue entry"));
> > + abort();
>
> Don't think we use abort() in libpq like that. There's some Assert()s
> tho.
>
>
For out-of-memory cases, we do abort(), and here abort will happen only
during some memory corruption. So, I think it is ok to abort here. Please
let me know if you think otherwise.
>
> > static bool
> > PQsendQueryStart(PGconn *conn)
> .....
>
Weirdly indented.
>
>
Corrected.
> > +/*
> > + * PQbatchBegin
>
> Mismatched w/ actual function name.
>
>
Corrected.
>
> > + * Put an idle connection in batch mode. Commands submitted after this
> > + * can be pipelined on the connection, there's no requirement to wait
> for
> > + * one to finish before the next is dispatched.
> > + *
> > + * Queuing of new query or syncing during COPY is not allowed.
>
> +"a"?
>
Hmm, Can you explain the question please. I don't understand.
>
> > +/*
> > + * PQbatchEnd
>
> wrong name.
>
Corrected.
>
> > + * End batch mode and return to normal command mode.
> > + *
> > + * Has no effect unless the client has processed all results
> > + * from all outstanding batches and the connection is idle.
>
> That seems wrong - will lead to hard to diagnose errors.
I have now added a error message when the batch end is failed to ease the
error diagnose. And, this behavior is documented. So the application should
not assume that the batch mode ended without checking the return value.
> > + * Returns true if batch mode ended.
> > + */
> > +int
> > +PQexitBatchMode(PGconn *conn)
> > +{
> > + if (!conn)
> > + return false;
> > +
> > + if (conn->batch_status == PQBATCH_MODE_OFF)
> > + return true;
> > +
> > + switch (conn->asyncStatus)
> > + {
> > + case PGASYNC_IDLE:
> > + printfPQExpBuffer(&conn->errorMessage,
> > + libpq_gettext_noop("internal error,
> IDLE in batch mode"));
> > + break;
> > + case PGASYNC_COPY_IN:
> > + case PGASYNC_COPY_OUT:
> > + case PGASYNC_COPY_BOTH:
> > + printfPQExpBuffer(&conn->errorMessage,
> > + libpq_gettext_noop("internal error,
> COPY in batch mode"));
> > + break;
>
> So we'll still check the queue in this case, that's a bit weird?
>
Removed above cases as adding error here is not of much help as well.
>
> > +/*
> > + * PQbatchQueueSync
>
> Out of sync.
>
Corrected.
>
> > + * End a batch submission by sending a protocol sync. The connection
> will
> > + * remain in batch mode and unavailable for new non-batch commands
> until all
> > + * results from the batch are processed by the client.
>
> "unavailable for new non-batch commands" - that's hard to follow, and
> seems pretty redundant with PQendBatchMode (or however it's called).
>
>
Modified documentation to be more precise and difference between
PQexitBatchMode and this one is that to exit batch, application should have
consumed all the results pending in buffer and post exit batch, an
application can call synchronous command execution functions. Whereas with
batch sync/commit, application cannot issue synchronous commands but can
issue asynchronous commands without reading all the results pending in
buffer. This function basically marks the end of implicit transaction.
> > + * It's legal to start submitting another batch immediately, without
> waiting
> > + * for the results of the current batch. There's no need to end batch
> mode
> > + * and start it again.
> > + *
> > + * If a command in a batch fails, every subsequent command up to and
> including
> > + * the PQbatchQueueSync command result gets set to
> PGRES_BATCH_ABORTED state. If the
> > + * whole batch is processed without error, a PGresult with
> PGRES_BATCH_END is
> > + * produced.
>
> Hm, should probably mention that that's only true for commands since the
> last PQbatchQueueSync?
>
>
"Batch" means the set of commands between PQbatchCommitQueue(newly renamed
name) calls except first batch which count from beginning to
PQbatchCommitQueue. And in documentation it is mentioned that "The result
for <function>PQbatchSyncQueue</function> is reported as
<literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
and resumption of normal result processing."
>
> > +/*
> > + * PQbatchQueueProcess
>
> Out of sync.
>
Corrected .
> > + * Returns false if the current query isn't done yet, the connection
> > + * is not in a batch, or there are no more queries to process.
>
> Last complaint about this - think this forgiving mode is a mistake.
>
>
Now added an error message using printfPQExpBuffer(). I think aborting here
is not really needed. Please let me know if you prefer to abort() rather
than this solution.
>
> > + */
> > +int
> > +PQbatchProcessQueue(PGconn *conn)
> > +{
>
> > + /* This command's results will come in immediately.
> > + * Initialize async result-accumulation state */
> > + pqClearAsyncResult(conn);
>
> I'm not following?
>
>
First line of comment is removed. It is copy/paste error.
>
> > /*
> > * PQgetResult
> > @@ -1749,10 +2228,32 @@ PQgetResult(PGconn *conn)
>
> > + if (conn->batch_status != PQBATCH_MODE_OFF)
> > + {
> > + /*
> > + * batched queries aren't followed by a
> Sync to put us back in
> > + * PGASYNC_IDLE state, and when we do get
> a sync we could
> > + * still have another batch coming after
> this one.
>
> This needs rephrasing.
>
>
Rephrased as "In batch mode, query execution state cannot be IDLE as there
can be other queries or results waiting in the queue ..." . Hope this is
simple and enough.
> > + * The connection isn't idle since we
> can't submit new
> > + * nonbatched commands. It isn't also busy
> since the current
> > + * command is done and we need to process
> a new one.
> > + */
> > + conn->asyncStatus = PGASYNC_QUEUED;
>
> Not sure I like the name.
>
>
Changed the name to PGASYNC_BATCH to make it more align to batch mode as
this status will not be set anywhere in non-batch mode.
>
> > + if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status !=
> PQBATCH_MODE_OFF)
> > + {
> > + printfPQExpBuffer(&conn->errorMessage,
> > +
> libpq_gettext("Synchronous command execution functions are not allowed in
> batch mode\n"));
> > + return false;
> > + }
>
> Why do we need the PGASYNC_QUEUED test here?
>
Yes, we don't need this check here. It was removed in PQfn and similarly it
is not needed here and in pqParseInput2 too.
> > +static int
> > +pqBatchFlush(PGconn *conn)
> > +{
> > + if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->
> outCount>=65536))
> > + return(pqFlush(conn));
> > + return 0; /* Just to keep compiler quiet */
> > +}
>
> This should be defined in a macro or such, rather than hardcoded.
>
>
Added macro "OUTBUFFER_THRESHOLD"
>
> Falling over now. This seems like enough feedback for a bit of work
> anyway.
>
Thanks once again for reviewing the patch. Attached the patch with your
review comments incorporation.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v13.patch (application/octet-stream, 52.1K) download
From f1c14fbf4368feb514126c9a82f2c33e59c011d8 Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Wed, 13 Sep 2017 14:08:25 +1000
Subject: [PATCH] Pipelining-batch-support-for-libpq-code-v13
---
doc/src/sgml/libpq.sgml | 502 +++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 5 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 595 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 47 +-
10 files changed, 1186 insertions(+), 43 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 096a8be..5e8a2e6 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4726,6 +4726,500 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchCommitQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchCommitQueue"><function>PQbatchCommitQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchCommitQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchCommitQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchCommitQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchCommitQueue">
+ <term>
+ <function>PQbatchCommitQueue</function>
+ <indexterm>
+ <primary>PQbatchCommitQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchCommitQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4766,6 +5260,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 3957bd3..00c6340 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -944,6 +944,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0..f907baa 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,8 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQenterBatchMode 173
+PQexitBatchMode 174
+PQbatchCommitQueue 175
+PQbatchProcessQueue 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index c580d91..8e043f7 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3566,6 +3566,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3577,6 +3596,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3608,6 +3628,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index c24bce6..d6a5f13 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -40,7 +40,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -71,7 +73,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1159,7 +1164,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1182,6 +1187,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1280,6 +1292,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1338,31 +1354,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1410,7 +1446,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1428,20 +1537,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_BATCH:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1465,6 +1614,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1474,6 +1627,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1586,35 +1756,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1741,6 +1918,280 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_BATCH;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ pqHandleSendFailure(conn);
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+/*
+ * PQbatchCommitQueue
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new synchronous command execution
+ * functions until all results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchCommitQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_BATCH:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return 0;
+}
+
+/*
+ * PQbatchProcessQueue
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_BATCH:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
/*
* PQgetResult
@@ -1800,10 +2251,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_BATCH:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_BATCH;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1983,6 +2455,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2177,6 +2656,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2192,6 +2674,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2200,15 +2696,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2218,14 +2717,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2620,6 +3123,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3813,3 +4323,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 83f74f3..4104cfe 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 7da5fb2..bd99287 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 1d915e7..7eb465a 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchCommitQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 4291360..1d47077 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_BATCH /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -385,6 +407,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -675,6 +708,12 @@ extern ssize_t pgtls_write(PGconn *conn, const void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold is set to 64k as it is safe
+ * in Windows as per comments in pqSendSome() API.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -683,6 +722,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-09-13 05:33 ` Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2017-09-13 05:33 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 13 September 2017 at 13:06, Vaishnavi Prabakaran <
vaishnaviprabakaran@gmail.com> wrote:
>
>
> On Wed, Aug 23, 2017 at 7:40 PM, Andres Freund <andres@anarazel.de> wrote:
>
>>
>>
>>
>> > Am failing to see the benefit in allowing user to set
>> > PQBatchAutoFlush(true|false) property? Is it really needed?
>>
>> I'm inclined not to introduce that for now. If somebody comes up with a
>> convincing usecase and numbers, we can add it later. Libpq API is set in
>> stone, so I'd rather not introduce unnecessary stuff...
>>
>>
> Thanks for reviewing the patch and yes ok.
>
>
>>
>>
>> > + <para>
>> > + Much like asynchronous query mode, there is no performance
>> disadvantage to
>> > + using batching and pipelining. It increases client application
>> complexity
>> > + and extra caution is required to prevent client/server deadlocks
>> but
>> > + can sometimes offer considerable performance improvements.
>> > + </para>
>>
>> That's not necessarily true, is it? Unless you count always doing
>> batches of exactly size 1.
>>
>
> Client application complexity is increased in batch mode,because
> application needs to remember the query queue status. Results processing
> can be done at anytime, so the application needs to know till what query,
> the results are consumed.
>
>
Yep. Also, the client/server deadlocks at issue here are a buffer
management issue, and deadlock is probably not exactly the right word. Your
app has to process replies from the server while it's sending queries,
otherwise it can get into a state where it has no room left in its send
buffer, but the server isn't consuming its receive buffer because the
server's send buffer is full. To allow the system to make progress, the
client must read from the client receive buffer.
This isn't an issue when using libpq normally.
PgJDBC has similar issues with its batch mode, but in PgJDBC it's much
worse because there's no non-blocking send available. In libpq you can at
least set your sending socket to non-blocking.
>
> > + <para>
>> > + Use batches when your application does lots of small
>> > + <literal>INSERT</literal>, <literal>UPDATE</literal> and
>> > + <literal>DELETE</literal> operations that can't easily be
>> transformed into
>> > + operations on sets or into a
>> > + <link linkend="libpq-copy"><literal>COPY</literal></link>
>> operation.
>> > + </para>
>>
>> Aren't SELECTs also a major beneficiarry of this?
>>
>
Yes, many individual SELECTs that cannot be assembled into a single more
efficient query would definitely also benefit.
> Hmm, though SELECTs also benefit from batch mode, doing multiple selects
> in batch mode will fill up the memory rapidly and might not be as
> beneficial as other operations listed.
>
Depends on the SELECT. With wide results you'll get less benefit, but even
then you can gain if you're on a high latency network. With "n+1" patterns
and similar, you'll see huge gains.
> Maybe note that multiple batches can be "in flight"?
>> I.e. PQbatchSyncQueue() is about error handling, nothing else? Don't
>> have a great idea, but we might want to rename...
>>
>>
> This function not only does error handling, but also sends the "Sync"
> message to backend. In batch mode, "Sync" message is not sent with every
> query but will
> be sent only via this function to mark the end of implicit transaction.
> Renamed it to PQbatchCommitQueue. Kindly let me know if you think of any
> other better name.
>
I really do not like calling it "commit" as that conflates with a database
commit.
A batch can embed multiple BEGINs and COMMITs. It's entirely possible for
an earlier part of the batch to succeed and commit, then a later part to
fail, if that's the case. So that name is IMO wrong.
>>
>> > + <varlistentry id="libpq-PQbatchSyncQueue">
>> > + <term>
>> > + <function>PQbatchSyncQueue</function>
>> > + <indexterm>
>> > + <primary>PQbatchSyncQueue</primary>
>> > + </indexterm>
>> > + </term>
>>
>> I wonder why this isn't framed as PQbatchIssue/Send/...()? Syncing seems
>> to mostly make sense from a protocol POV.
>>
>>
> Renamed to PQbatchCommitQueue.
>
>
Per above, strong -1 on that. But SendQueue seems OK, or FlushQueue?
>
>> > + * Put an idle connection in batch mode. Commands submitted after
>> this
>> > + * can be pipelined on the connection, there's no requirement to
>> wait for
>> > + * one to finish before the next is dispatched.
>> > + *
>> > + * Queuing of new query or syncing during COPY is not allowed.
>>
>> +"a"?
>>
>
> Hmm, Can you explain the question please. I don't understand.
>
s/of new query/of a new query/
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2017-09-13 05:44 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 06:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
0 siblings, 2 replies; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-09-13 05:44 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Andres Freund <andres@anarazel.de>; Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Wed, Sep 13, 2017 at 3:33 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
>
> I really do not like calling it "commit" as that conflates with a database
> commit.
>
> A batch can embed multiple BEGINs and COMMITs. It's entirely possible for
> an earlier part of the batch to succeed and commit, then a later part to
> fail, if that's the case. So that name is IMO wrong.
>
Ok, SendQueue seems ok to me as well. Will change it in next version.
>>> +"a"?
>>>
>>
>> Hmm, Can you explain the question please. I don't understand.
>>
>
> s/of new query/of a new query/
>
>
Thanks for explaining. Will change this too in next version.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-09-13 06:01 ` Craig Ringer <craig@2ndquadrant.com>
1 sibling, 0 replies; 182+ messages in thread
From: Craig Ringer @ 2017-09-13 06:01 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On 13 September 2017 at 13:44, Vaishnavi Prabakaran <
vaishnaviprabakaran@gmail.com> wrote:
> Thanks for explaining. Will change this too in next version.
>
>
Thankyou, a lot, for picking up this patch.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-10-02 09:31 ` Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
1 sibling, 1 reply; 182+ messages in thread
From: Daniel Gustafsson @ 2017-10-02 09:31 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Craig Ringer <craig@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
> On 13 Sep 2017, at 07:44, Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com> wrote:
>
> On Wed, Sep 13, 2017 at 3:33 PM, Craig Ringer <craig@2ndquadrant.com <mailto:craig@2ndquadrant.com>> wrote:
>
> I really do not like calling it "commit" as that conflates with a database commit.
>
> A batch can embed multiple BEGINs and COMMITs. It's entirely possible for an earlier part of the batch to succeed and commit, then a later part to fail, if that's the case. So that name is IMO wrong.
>
> Ok, SendQueue seems ok to me as well. Will change it in next version.
>
> +"a"?
>
> Hmm, Can you explain the question please. I don't understand.
>
> s/of new query/of a new query/
>
> Thanks for explaining. Will change this too in next version.
Based on the discussions in this thread, and that a new version hasn’t been
submitted, I’m marking this Returned with Feedback. Please re-submit the new
version in an upcoming commitfest when ready.
cheers ./daniel
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
@ 2017-10-05 00:58 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2017-10-05 00:58 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: Craig Ringer <craig@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Mon, Oct 2, 2017 at 8:31 PM, Daniel Gustafsson <daniel@yesql.se> wrote:
> > On 13 Sep 2017, at 07:44, Vaishnavi Prabakaran <
> vaishnaviprabakaran@gmail.com> wrote:
> >
> > On Wed, Sep 13, 2017 at 3:33 PM, Craig Ringer <craig@2ndquadrant.com
> <mailto:craig@2ndquadrant.com>> wrote:
> >
> > I really do not like calling it "commit" as that conflates with a
> database commit.
> >
> > A batch can embed multiple BEGINs and COMMITs. It's entirely possible
> for an earlier part of the batch to succeed and commit, then a later part
> to fail, if that's the case. So that name is IMO wrong.
> >
> > Ok, SendQueue seems ok to me as well. Will change it in next version.
> >
> > +"a"?
> >
> > Hmm, Can you explain the question please. I don't understand.
> >
> > s/of new query/of a new query/
> >
> > Thanks for explaining. Will change this too in next version.
>
> Based on the discussions in this thread, and that a new version hasn’t been
> submitted, I’m marking this Returned with Feedback. Please re-submit the
> new
> version in an upcoming commitfest when ready.
Thanks for the suggestion and, OK I will create a new patch in upcoming
commitfest with attached patch addressing above review comments.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v14.patch (application/octet-stream, 52.1K) download
From 0b2fbd49c73f9783ffa09cb75c425d75852c4d56 Mon Sep 17 00:00:00 2001
From: Prabakaran <Vaishnavi.Prabakaran@au.fujitsu.com>
Date: Thu, 5 Oct 2017 11:55:50 +1100
Subject: [PATCH] Pipelining-batch-support-for-libpq-code-v14
---
doc/src/sgml/libpq.sgml | 502 +++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 5 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 595 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 47 +-
10 files changed, 1186 insertions(+), 43 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0aedd83..e353658 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4728,6 +4728,500 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4768,6 +5262,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 7757e1e..db8523d 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 3957bd3..00c6340 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -944,6 +944,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0..625e74e 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,8 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQenterBatchMode 173
+PQexitBatchMode 174
+PQbatchSendQueue 175
+PQbatchProcessQueue 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 5f79803..ee8c3b4 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3566,6 +3566,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3577,6 +3596,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3608,6 +3628,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index c24bce6..99c0b49 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -40,7 +40,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -71,7 +73,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1159,7 +1164,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1182,6 +1187,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1280,6 +1292,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1338,31 +1354,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1410,7 +1446,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1428,20 +1537,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1465,6 +1614,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1474,6 +1627,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1586,35 +1756,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1741,6 +1918,280 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ pqHandleSendFailure(conn);
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+/*
+ * PQbatchSendQueue
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new synchronous command execution
+ * functions until all results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return 0;
+}
+
+/*
+ * PQbatchProcessQueue
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
/*
* PQgetResult
@@ -1800,10 +2251,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1983,6 +2455,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2177,6 +2656,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2192,6 +2674,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2200,15 +2696,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2218,14 +2717,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2620,6 +3123,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3813,3 +4323,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 1320d18..7ce6885 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -411,6 +411,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 21fb8f2..7d05af3 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -219,10 +219,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -879,6 +887,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 1d915e7..98993fa 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 4291360..1d47077 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -385,6 +407,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -675,6 +708,12 @@ extern ssize_t pgtls_write(PGconn *conn, const void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold is set to 64k as it is safe
+ * in Windows as per comments in pqSendSome() API.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -683,6 +722,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2017-11-28 01:57 ` Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Michael Paquier @ 2017-11-28 01:57 UTC (permalink / raw)
To: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; +Cc: Daniel Gustafsson <daniel@yesql.se>; Craig Ringer <craig@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Daniel Verite <daniel@manitou-mail.org>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Thu, Oct 5, 2017 at 9:58 AM, Vaishnavi Prabakaran
<vaishnaviprabakaran@gmail.com> wrote:
> Thanks for the suggestion and, OK I will create a new patch in upcoming
> commitfest with attached patch addressing above review comments.
The patch still applies and there has been no updates for the last
month, as well as no reviews. I am bumping it to next CF.
--
Michael
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2018-01-05 05:55 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2018-01-05 05:55 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Daniel Gustafsson <daniel@yesql.se>; Craig Ringer <craig@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Daniel Verite <daniel@manitou-mail.org>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Tue, Nov 28, 2017 at 12:57 PM, Michael Paquier <michael.paquier@gmail.com
> wrote:
> On Thu, Oct 5, 2017 at 9:58 AM, Vaishnavi Prabakaran
> <vaishnaviprabakaran@gmail.com> wrote:
> > Thanks for the suggestion and, OK I will create a new patch in upcoming
> > commitfest with attached patch addressing above review comments.
>
> The patch still applies and there has been no updates for the last
> month, as well as no reviews. I am bumping it to next CF.
Thank you, I see the patch generates a compilation error due to usage of
"FALSE" with latest postgres code, Hence attaching the patch with
correction.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v15.patch (application/octet-stream, 52.1K) download
From 40208e16e478d0a95bb81b7a28dfe8332ddd8b94 Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <vaishnavip@fast.au.fujitsu.com>
Date: Fri, 5 Jan 2018 16:52:02 +1100
Subject: [PATCH] Pipelining-batch-support-for-libpq-code-v15
---
doc/src/sgml/libpq.sgml | 502 +++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 5 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 595 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 47 +-
10 files changed, 1186 insertions(+), 43 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4e46451..b540e75 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4752,6 +4752,500 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending">.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4792,6 +5286,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 086cb8d..7de9e30 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 3957bd3..00c6340 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -944,6 +944,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0..625e74e 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,8 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQenterBatchMode 173
+PQexitBatchMode 174
+PQbatchSendQueue 175
+PQbatchProcessQueue 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 68fb9a1..037bda2 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3574,6 +3574,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3585,6 +3604,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3616,6 +3636,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 6653087..961ff41 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -40,7 +40,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -71,7 +73,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1159,7 +1164,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1182,6 +1187,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1280,6 +1292,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1338,31 +1354,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1410,7 +1446,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1428,20 +1537,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1465,6 +1614,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1474,6 +1627,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1586,35 +1756,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1741,6 +1918,280 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ pqHandleSendFailure(conn);
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+/*
+ * PQbatchSendQueue
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new synchronous command execution
+ * functions until all results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return 0;
+}
+
+/*
+ * PQbatchProcessQueue
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
/*
* PQgetResult
@@ -1800,10 +2251,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1983,6 +2455,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2177,6 +2656,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2192,6 +2674,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2200,15 +2696,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2218,14 +2717,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2620,6 +3123,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3813,3 +4323,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 5335a91..eac7714 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -411,6 +411,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 0c5099c..be64000 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -219,10 +219,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -879,6 +887,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 1d915e7..98993fa 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index f6c1023..edf4a7a 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -386,6 +408,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -397,6 +420,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -679,6 +712,12 @@ extern char *pgtls_get_finished(PGconn *conn, size_t *len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold is set to 64k as it is safe
+ * in Windows as per comments in pqSendSome() API.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -687,6 +726,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2018-01-11 23:12 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
0 siblings, 1 reply; 182+ messages in thread
From: Vaishnavi Prabakaran @ 2018-01-11 23:12 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Daniel Gustafsson <daniel@yesql.se>; Craig Ringer <craig@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Daniel Verite <daniel@manitou-mail.org>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
On Fri, Jan 5, 2018 at 4:55 PM, Vaishnavi Prabakaran <
vaishnaviprabakaran@gmail.com> wrote:
>
>
> On Tue, Nov 28, 2017 at 12:57 PM, Michael Paquier <
> michael.paquier@gmail.com> wrote:
>
>> On Thu, Oct 5, 2017 at 9:58 AM, Vaishnavi Prabakaran
>> <vaishnaviprabakaran@gmail.com> wrote:
>> > Thanks for the suggestion and, OK I will create a new patch in upcoming
>> > commitfest with attached patch addressing above review comments.
>>
>> The patch still applies and there has been no updates for the last
>> month, as well as no reviews. I am bumping it to next CF.
>
>
> Thank you, I see the patch generates a compilation error due to usage of
> "FALSE" with latest postgres code, Hence attaching the patch with
> correction.
>
Corrected compilation error in documentation portion of patch with latest
postgres code. Attached the corrected patch.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
Attachments:
0001-Pipelining-batch-support-for-libpq-code-v16.patch (application/octet-stream, 52.1K) download
From ba93ae02eca024997f2ce6a9c2c2987aea4a77b8 Mon Sep 17 00:00:00 2001
From: Prabakaran <Vaishnavi.Prabakaran@au.fujitsu.com>
Date: Fri, 12 Jan 2018 10:09:09 +1100
Subject: [PATCH] Pipelining-batch-support-for-libpq-code-v16
---
doc/src/sgml/libpq.sgml | 502 +++++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 5 +
src/interfaces/libpq/fe-connect.c | 28 +
src/interfaces/libpq/fe-exec.c | 595 +++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 47 +-
10 files changed, 1186 insertions(+), 43 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4e46451..6aae637 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4752,6 +4752,500 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending"/>.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode"/>.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4792,6 +5286,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <function>PQclear</function> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-pqsetsinglerowmode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6b5aaeb..cf22f3f 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index f9aec05..32151ff 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -944,6 +944,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index d6a38d0..625e74e 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -172,3 +172,8 @@ PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
PQencryptPasswordConn 172
+PQenterBatchMode 173
+PQexitBatchMode 174
+PQbatchSendQueue 175
+PQbatchProcessQueue 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 8d54333..c2df669 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3574,6 +3574,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3585,6 +3604,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3616,6 +3636,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 4c0114c..2049035 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -40,7 +40,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -71,7 +73,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1159,7 +1164,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1182,6 +1187,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1280,6 +1292,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1338,31 +1354,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1410,7 +1446,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1428,20 +1537,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1465,6 +1614,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1474,6 +1627,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1586,35 +1756,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1741,6 +1918,280 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ pqHandleSendFailure(conn);
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+/*
+ * PQbatchSendQueue
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new synchronous command execution
+ * functions until all results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return 0;
+}
+
+/*
+ * PQbatchProcessQueue
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
/*
* PQgetResult
@@ -1800,10 +2251,31 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1983,6 +2455,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2177,6 +2656,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2192,6 +2674,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2200,15 +2696,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2218,14 +2717,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -2620,6 +3123,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3813,3 +4323,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 7dcef80..26c9a9e 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -411,6 +411,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index d3ca5d2..f699607 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -219,10 +219,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -879,6 +887,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index ed9c806..7136980 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 4e35409..fc5376f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -386,6 +408,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -397,6 +420,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -680,6 +713,12 @@ extern char *pgtls_get_peer_certificate_hash(PGconn *conn, size_t *len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold is set to 64k as it is safe
+ * in Windows as per comments in pqSendSome() API.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -688,6 +727,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
@ 2018-03-22 12:11 ` Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
0 siblings, 1 reply; 182+ messages in thread
From: Kyotaro HORIGUCHI @ 2018-03-22 12:11 UTC (permalink / raw)
To: vaishnaviprabakaran@gmail.com; +Cc: michael.paquier@gmail.com; daniel@yesql.se; craig@2ndquadrant.com; andres@anarazel.de; daniel@manitou-mail.org; david@pgmasters.net; VaishnaviP@fast.au.fujitsu.com; kommi.haribabu@gmail.com; tsunakawa.takay@jp.fujitsu.com; dmitigr@gmail.com; pgsql-hackers; m.kniep@web.de; fujita.etsuro@lab.ntt.co.jp; iwata.aya@jp.fujitsu.com
Hello.
At Fri, 12 Jan 2018 10:12:35 +1100, Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com> wrote in <CAOoUkxRRTBm8ztzVXL45EVk=BC8AjfaFM5=ZMtrOMMfm+dnbwA@mail.gmail.com>
> Corrected compilation error in documentation portion of patch with latest
> postgres code. Attached the corrected patch.
My understanding of this patch is that it intends to do the
following things without changing the protocol.
A. Refrain from sending sync message and socket flush during batching.
B. Set required information to PGconn before receiving a result.
Multi statement query does the a similar thing but a bit
different. Queries are not needed to be built and mreged as a
string in advance with this feature.
Since I'm not sure what is the current stage of this patch and I
haven't fully recognize the disucssion so far, I might make
stupid or duplicate comments but would like to make some
comments.
- Previous comments
A disucssion on psql batch mode was held in another branch of
this thread. How do we treat that?
- Interface complteness
I think PQenter/exitBatchMode() is undoubtedly needed.
PQbatchSendQueue() seems to be required to make sure to send
queries before calling PQgetResult.
PQbatchProcessQueue() doesn't seem essential. If there's no case
where it is called without following PQgetResult, it can be
included in PQgetResult. Client code may be simpler if it is not
required.
The latest patch has extern definition of PQbatchQueueCount, the
body and doc of which seems to have been removed, or haven't
exist since the beginning.
- Some comments on the code
-- PQbatchProcessQueue()
PQbatchProcessQueue() returns 0 for several undistinguishable
reasons. If asyncStatus is PGASYNC_BUSY at the time, the
connection is out of sync and doesn't accept further
operations. So it should be distinguished from the end-of-queue
case. (It can reutrn an error result if combining with
PQgetResult.)
The function handles COPY_* in inconsistent way. It sets an error
message to conn in hardly noticeable way but allows to go
further. The document is telling as the follows.
> COPY is not recommended as it most likely will trigger failure
> in batch processing
I don't think the desciription is informative, but the reason for
the description would be the fact that we cannot detect a command
leads to protocol failure before the failure actually
happens. The test code suggests that that is the case where any
command (other than Sync) following "COPY FROM" breaks the
protocol. We get an error messages like below in the case. (The
header part is of my test program.)
> ABORTED: (PGRES_BATCH_ABORTED) ERROR: unexpected message type 0x42 during COPY from stdin
> CONTEXT: COPY t, line 1
I haven't investigated further, but it seems unable to restore
the sync of connection until disconnection. It would need a
side-channel for COPY_IN to avoid the failure but we don't have
it now. The possible choices here are:
A. Make the description more precisely, by adding the condition
to cause the error and how it looks.
B. The situation seems detectable in PGgetResult. Return an error
result containing any meaningful error. But this doesn't stop
the following protocol error messages.
> ABORTED: (PGRES_FATAL_ERROR) COPY_IN cannot be followed by any command in a batch
> ABORTED: (PGRES_FATAL_ERROR) ERROR: 08P01: unexpected message type 0x42 during COPY from stdin
> ABORTED: (PGRES_BATCH_ABORTED) ERROR: unexpected message type 0x42 during COPY from stdin
> ABORTED: (PGRES_BATCH_ABORTED) ERROR: unexpected message type 0x42 during COPY from stdin
C. Cancel COPY on the server and return error to the client for
the situation. This might let us be able to avoid
unrecoverable desync of the protocol in the case. (I'm not
sure this can be done cleanly.)
D. Wait for a protocol change that can solve this problem.
-- PQbatchFlush()
> static int pqBatchFlush(Pgconn *conn)
...
> if (...)
> return(pqFlush(conn));
> return 0; /* Just to keep compiler quiet */
The comment in the last line is wrong.
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_T
Isn't this a bit too long?
-- PGcommandQueueEntry
The resason for the fact that it is a linked list seems to me to
allow queueing and result processing happen alternately in one
batch period. But I coundn't find a description about the
behavior in the documentation. (State transition chart is
required?)
Aside from the above, the interface to handle the command queue
seems to be divided into too-small pieces.
PQsendQueryGuts()
> el = PQmakePipelinedCommand(conn)
> el->members = hoge;
> ..
> PQappendPipelinedCommand(conn, el)
Isn't the following enough instead of the above?
> PQappendPipelinedCommand(conn, conn->queryclass, conn->last_query);
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
@ 2018-03-23 13:18 ` Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2018-03-23 13:18 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; +Cc: vaishnaviprabakaran@gmail.com; michael.paquier@gmail.com; daniel@yesql.se; craig@2ndquadrant.com; andres@anarazel.de; david@pgmasters.net; VaishnaviP@fast.au.fujitsu.com; kommi.haribabu@gmail.com; tsunakawa.takay@jp.fujitsu.com; dmitigr@gmail.com; pgsql-hackers; m.kniep@web.de; fujita.etsuro@lab.ntt.co.jp; iwata.aya@jp.fujitsu.com
Kyotaro HORIGUCHI wrote:
> A disucssion on psql batch mode was held in another branch of
> this thread. How do we treat that?
There's a batch mode for pgbench in a patch posted in [1],
with \beginbatch and \endbatch commands, but nothing
for psql AFAICS.
psql is more complicated because currently it uses a
blocking PQexec() call at its core. Craig mentioned psql
integration in [2] and [3].
Also a script can have inter-query dependencies such as in
insert into table(...) returning id \gset
update othertable set col= :id where ...;
which is a problem in batch mode, as we don't want
to send the update before the right value for :id is
known. Whether we want to support these dependencies
and how needs discussion.
For instance we might not support them at all, or
create a synchronization command that collects
all results of queries sent so far, or do it implicitly
when a variable is injected into a query...
This looks like substantial work that might be best
done separately from the libpq patch.
[1]
https://www.postgresql.org/message-id/b4e34135-2bd9-4b8a-94ca-27d760da26d7@manitou-mail.org
[2]
https://www.postgresql.org/message-id/CAMsr+YGLhaDkjymLuNVQy4MrSKQoA=F1vO=aN8XQf30N=aQuVA@mail.gmail...
[3]
https://www.postgresql.org/message-id/CAMsr+YE6BK4iAaQz=nY3xDnbLhnNZ_4tp-PTJqbNNpsZMgoo8Q@mail.gmail...
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2018-10-01 06:34 ` Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Michael Paquier @ 2018-10-01 06:34 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; vaishnaviprabakaran@gmail.com; michael.paquier@gmail.com; daniel@yesql.se; craig@2ndquadrant.com; andres@anarazel.de; david@pgmasters.net; VaishnaviP@fast.au.fujitsu.com; kommi.haribabu@gmail.com; tsunakawa.takay@jp.fujitsu.com; dmitigr@gmail.com; pgsql-hackers; m.kniep@web.de; fujita.etsuro@lab.ntt.co.jp; iwata.aya@jp.fujitsu.com
On Fri, Mar 23, 2018 at 02:18:09PM +0100, Daniel Verite wrote:
> There's a batch mode for pgbench in a patch posted in [1],
> with \beginbatch and \endbatch commands, but nothing
> for psql AFAICS.
> psql is more complicated because currently it uses a
> blocking PQexec() call at its core. Craig mentioned psql
> integration in [2] and [3].
This patch has been around for some time now, the last version fails to
apply cleanly and in-depth reviews have happened. I am moving that to
the next CF, waiting on its author.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
@ 2018-11-29 16:14 ` Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Dmitry Dolgov @ 2018-11-29 16:14 UTC (permalink / raw)
To: vaishnaviprabakaran@gmail.com, Craig Ringer <craig@2ndquadrant.com>; +Cc: daniel@manitou-mail.org, horiguchi.kyotaro@lab.ntt.co.jp, Michael Paquier <michael.paquier@gmail.com>; daniel@yesql.se, Andres Freund <andres@anarazel.de>; David Steele <david@pgmasters.net>; VaishnaviP@fast.au.fujitsu.com, Haribabu Kommi <kommi.haribabu@gmail.com>; pgsql-hackers; m.kniep@web.de, fujita.etsuro@lab.ntt.co.jp, iwata.aya@jp.fujitsu.com, Michael Paquier <michael@paquier.xyz>
> On Mon, Oct 1, 2018 at 8:34 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Fri, Mar 23, 2018 at 02:18:09PM +0100, Daniel Verite wrote:
> > There's a batch mode for pgbench in a patch posted in [1],
> > with \beginbatch and \endbatch commands, but nothing
> > for psql AFAICS.
> > psql is more complicated because currently it uses a
> > blocking PQexec() call at its core. Craig mentioned psql
> > integration in [2] and [3].
>
> This patch has been around for some time now, the last version fails to
> apply cleanly and in-depth reviews have happened. I am moving that to
> the next CF, waiting on its author.
Unfortunately, nothing was changed since then, so there is already some amount
of unaddressed review feedback. I'll move this patch to "Returned with
feedback".
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
@ 2019-08-30 13:36 ` Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Nikhil Sontakke @ 2019-08-30 13:36 UTC (permalink / raw)
To: Dmitry Dolgov <9erthalion6@gmail.com>; +Cc: vaishnaviprabakaran@gmail.com, Craig Ringer <craig@2ndquadrant.com>; daniel@manitou-mail.org, horiguchi.kyotaro@lab.ntt.co.jp, Michael Paquier <michael.paquier@gmail.com>; daniel@yesql.se, Andres Freund <andres@anarazel.de>; David Steele <david@pgmasters.net>; VaishnaviP@fast.au.fujitsu.com, Haribabu Kommi <kommi.haribabu@gmail.com>; pgsql-hackers; m.kniep@web.de, fujita.etsuro@lab.ntt.co.jp, iwata.aya@jp.fujitsu.com, Michael Paquier <michael@paquier.xyz>
Hi,
> > This patch has been around for some time now, the last version fails to
> > apply cleanly and in-depth reviews have happened. I am moving that to
> > the next CF, waiting on its author.
>
> Unfortunately, nothing was changed since then, so there is already some amount
> of unaddressed review feedback. I'll move this patch to "Returned with
> feedback".
>
Craig Ringer mentioned about this thread to me recently.
This effort has seen decent reviews from Craig, Andres and Michael
already. So, I thought of refreshing it to work against latest master
HEAD.
PFA, main patch as well as the test patch (I named the test patch v17
to be consistent with the main patch). The major grouse with the test
patch AFAICS was the use of non-Windows compliant timersub() function.
I have now used INSTR_TIME_SET_CURRENT/INSTR_TIME_SUBTRACT family of
portable macros for the same.
Please let me know on what we think of the above.
Regards,
Nikhil
--
Nikhil Sontakke
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
Attachments: download all (2 files, .tar.gz)
0002-libpq_batch_tests_community_master.v17.patch (application/octet-stream, 38.8K) download
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 953905d1ef..6eeec4f397 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -20,6 +20,7 @@ SUBDIRS = \
test_rbtree \
test_rls_hooks \
test_shm_mq \
+ test_libpq \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..11e8463984
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..d907063f65
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..25f9ff0f89
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,25 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000000..bcebde1a01
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1456 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "c.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchProcessQueue when there might still be pending results */
+ if (PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSendQueue(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ instr_time start_time, end_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_pipelined(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+
+ printf("batch insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_sequential(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("sequential insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_copy(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("COPY elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQenterBatchMode(conn);
+
+ for (i=0; i < 3; i++) {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchSendQueue(conn);
+
+ i=0;
+ while (PQbatchProcessQueue(conn))
+ {
+ int isSingleTuple = 0;
+ /* Set single row mode for only first 3 SELECT queries */
+ if(i < 3)
+ r = PQsetSingleRowMode(conn);
+ if (r!=1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ if(!isSingleTuple)
+ {
+ fprintf(stderr, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ goto fail;
+ }
+ isSingleTuple=0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ }
+ i++;
+ }
+ PQexitBatchMode(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if(run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 2eab635898..7c67fc9330 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -48,7 +48,8 @@ my @contrib_excludes = (
'ltree_plpython', 'pgcrypto',
'sepgsql', 'brin',
'test_extensions', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'snapshot_too_old', 'unsafe_tests',
+ 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
0001-libpq_batch_support_commmunity_master.v17.patch (application/octet-stream, 51.5K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index b7c3d96b01..50d4cd279c 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4663,6 +4663,500 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending"/>.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode"/>.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
@@ -4703,6 +5197,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 34dcfabe73..6f527267e0 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6eba08a920..d41164edfa 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -984,6 +984,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 7c808e5215..ffcbe50c61 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -176,3 +176,8 @@ PQresultMemorySize 173
PQhostaddr 174
PQgssEncInUse 175
PQgetgssctx 176
+PQenterBatchMode 177
+PQexitBatchMode 178
+PQbatchSendQueue 179
+PQbatchProcessQueue 180
+PQbatchStatus 181
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 7f1fd2f45e..1a9a552e53 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -502,6 +502,24 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
/*
* pqDropServerData
@@ -521,6 +539,7 @@ pqDropServerData(PGconn *conn)
{
PGnotify *notify;
pgParameterStatus *pstatus;
+ PGcommandQueueEntry *queue;
/* Forget pending notifies */
notify = conn->notifyHead;
@@ -533,6 +552,14 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index b3c59a0992..db79015100 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -40,7 +40,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -71,7 +73,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1211,7 +1216,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1234,6 +1239,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1332,6 +1344,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1390,31 +1406,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1462,7 +1498,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1480,20 +1589,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1517,6 +1666,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1526,6 +1679,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1638,35 +1808,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1767,6 +1944,280 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ /* error message should be set up already */
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+/*
+ * PQbatchSendQueue
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new synchronous command execution
+ * functions until all results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * PQbatchProcessQueue
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
/*
* PQgetResult
@@ -1843,9 +2294,30 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -2026,6 +2498,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2220,6 +2699,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2235,6 +2717,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2243,15 +2739,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2261,14 +2760,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2666,6 +3169,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3859,3 +4369,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 0e36974fc4..8357334027 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -411,6 +411,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index b04f7ec123..0cb1ac03f9 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -219,10 +219,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -879,6 +887,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 22c4954f2b..bb452f28c5 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -136,6 +139,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -434,6 +448,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index d37bb3ce40..eed6945cff 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -389,6 +411,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -401,6 +424,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -781,6 +814,12 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold is set to 64k as it is safe
+ * in Windows as per comments in pqSendSome() API.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -789,6 +828,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
@ 2019-09-09 07:20 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Amit Kapila @ 2019-09-09 07:20 UTC (permalink / raw)
To: Nikhil Sontakke <nikhils@2ndquadrant.com>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; vaishnaviprabakaran@gmail.com, Craig Ringer <craig@2ndquadrant.com>; daniel@manitou-mail.org, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; Michael Paquier <michael.paquier@gmail.com>; Daniel Gustafsson <daniel@yesql.se>; Andres Freund <andres@anarazel.de>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; pgsql-hackers; m.kniep@web.de, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>; iwata.aya@jp.fujitsu.com, Michael Paquier <michael@paquier.xyz>
On Fri, Aug 30, 2019 at 7:06 PM Nikhil Sontakke <nikhils@2ndquadrant.com> wrote:
>
> Hi,
>
> > > This patch has been around for some time now, the last version fails to
> > > apply cleanly and in-depth reviews have happened. I am moving that to
> > > the next CF, waiting on its author.
> >
> > Unfortunately, nothing was changed since then, so there is already some amount
> > of unaddressed review feedback. I'll move this patch to "Returned with
> > feedback".
> >
>
> Craig Ringer mentioned about this thread to me recently.
>
> This effort has seen decent reviews from Craig, Andres and Michael
> already. So, I thought of refreshing it to work against latest master
> HEAD.
>
Thanks for picking up this. However, I noticed that previously
Horiguchi-San has given some comments on this patch [1] which doesn't
seem to be addressed or at least not all of them are addressed. It is
possible that you would have already addressed those, but in that
case, it would be good if you respond to his email as well. If those
are not addressed, then it will be good to address those.
[1] - https://www.postgresql.org/message-id/20180322.211148.187821341.horiguchi.kyotaro%40lab.ntt.co.jp
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
@ 2020-07-10 23:01 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2020-07-10 23:01 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Nikhil Sontakke <nikhils@2ndquadrant.com>; Dmitry Dolgov <9erthalion6@gmail.com>; vaishnaviprabakaran@gmail.com, Craig Ringer <craig@2ndquadrant.com>; daniel@manitou-mail.org, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; Michael Paquier <michael.paquier@gmail.com>; Daniel Gustafsson <daniel@yesql.se>; Andres Freund <andres@anarazel.de>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; pgsql-hackers; m.kniep@web.de, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>; iwata.aya@jp.fujitsu.com, Michael Paquier <michael@paquier.xyz>
On 2019-Sep-09, Amit Kapila wrote:
> Thanks for picking up this. However, I noticed that previously
> Horiguchi-San has given some comments on this patch [1] which doesn't
> seem to be addressed or at least not all of them are addressed. It is
> possible that you would have already addressed those, but in that
> case, it would be good if you respond to his email as well. If those
> are not addressed, then it will be good to address those.
Totally unasked for, here's a rebase of this patch series. I didn't do
anything other than rebasing to current master, solving a couple of very
trivial conflicts, fixing some whitespace complaints by git apply, and
running tests to verify everthing works.
I don't foresee working on this at all, so if anyone is interested in
seeing this feature in, I encourage them to read and address
Horiguchi-san's feedback.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments: download all (2 files, .tar.gz)
v18-0001-libpq-batch-support.patch (text/x-diff, 52.3K) download
From c2de4e2a2cbc301b48a7946a9ee0b9acceae7233 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jul 2020 11:29:11 -0400
Subject: [PATCH v18 1/2] libpq batch support
---
doc/src/sgml/libpq.sgml | 502 +++++++++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 5 +
src/interfaces/libpq/fe-connect.c | 27 +
src/interfaces/libpq/fe-exec.c | 597 ++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 47 +-
10 files changed, 1186 insertions(+), 44 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index d1ccaa775a..ad168727c7 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4826,6 +4826,500 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending"/>.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode"/>.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4866,6 +5360,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index cf4653fe0f..a1b46c83ed 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9bb6..5cac751d99 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1001,6 +1001,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..ffb18fcf47 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,8 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterBatchMode 180
+PQexitBatchMode 181
+PQbatchSendQueue 182
+PQbatchProcessQueue 183
+PQbatchStatus 184
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 27c9bb46ee..0e665b3df4 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -529,6 +529,24 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
/*
* pqDropServerData
@@ -548,6 +566,7 @@ pqDropServerData(PGconn *conn)
{
PGnotify *notify;
pgParameterStatus *pstatus;
+ PGcommandQueueEntry *queue;
/* Forget pending notifies */
notify = conn->notifyHead;
@@ -560,6 +579,14 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index eea0237c3a..eb9b8fc8c5 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -70,7 +72,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1210,7 +1215,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1233,6 +1238,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1331,6 +1343,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1389,31 +1405,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1461,7 +1497,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1479,20 +1588,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1516,6 +1665,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1525,6 +1678,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1637,35 +1807,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1766,6 +1943,280 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ /* error message should be set up already */
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+/*
+ * PQbatchSendQueue
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new synchronous command execution
+ * functions until all results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * PQbatchProcessQueue
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
/*
* PQgetResult
@@ -1842,9 +2293,30 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -2025,6 +2497,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2219,6 +2698,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2234,6 +2716,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2242,15 +2738,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2260,14 +2759,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2665,6 +3168,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3858,3 +4368,14 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 9360c541be..2ff3fa4883 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -406,6 +406,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 1696525475..9c37a9c4ff 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -217,10 +217,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -875,6 +883,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 3b6a9fbce3..fc0e590985 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -97,7 +97,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -137,6 +140,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -435,6 +449,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1de91ae295..88d168f804 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -394,6 +416,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -406,6 +429,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -798,6 +831,12 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold is set to 64k as it is safe
+ * in Windows as per comments in pqSendSome() API.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -806,6 +845,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.20.1
v18-0002-libpq-batch-tests.patch (text/x-diff, 40.2K) download
From 47c4e9c0af9dc5abe6dd17101945ab4f10291958 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jul 2020 11:30:38 -0400
Subject: [PATCH v18 2/2] libpq batch tests
---
src/test/modules/Makefile | 1 +
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
.../modules/test_libpq/t/001_libpq_async.pl | 27 +
src/test/modules/test_libpq/testlibpqbatch.c | 1456 +++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 16 +-
7 files changed, 1523 insertions(+), 8 deletions(-)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 29de73c060..a4ba8c04c1 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -22,6 +22,7 @@ SUBDIRS = \
test_rbtree \
test_rls_hooks \
test_shm_mq \
+ test_libpq \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..11e8463984
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..d907063f65
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..2c361880fa
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,27 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_batch simple_batch multi_batch batch_abort timings singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'testlibpqbatch', 'dbname=postgres', "$numrows", "$testname" ],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000000..4d6ba266e5
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1456 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "c.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchProcessQueue when there might still be pending results */
+ if (PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+ goto fail;
+ }
+
+ if (PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchSendQueue(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQenterBatchMode(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchProcessQueue(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSendQueue(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQexitBatchMode(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ instr_time start_time, end_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_pipelined(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+
+ printf("batch insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_sequential(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("sequential insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_copy(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("COPY elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQenterBatchMode(conn);
+
+ for (i=0; i < 3; i++) {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchSendQueue(conn);
+
+ i=0;
+ while (PQbatchProcessQueue(conn))
+ {
+ int isSingleTuple = 0;
+ /* Set single row mode for only first 3 SELECT queries */
+ if(i < 3)
+ r = PQsetSingleRowMode(conn);
+ if (r!=1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ if(!isSingleTuple)
+ {
+ fprintf(stderr, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ goto fail;
+ }
+ isSingleTuple=0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ }
+ i++;
+ }
+ PQexitBatchMode(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if(run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 20da7985c1..0054a394ea 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -43,14 +43,14 @@ my $contrib_extrasource = {
'seg' => [ 'contrib/seg/segscan.l', 'contrib/seg/segparse.y' ],
};
my @contrib_excludes = (
- 'bool_plperl', 'commit_ts',
- 'hstore_plperl', 'hstore_plpython',
- 'intagg', 'jsonb_plperl',
- 'jsonb_plpython', 'ltree_plpython',
- 'pgcrypto', 'sepgsql',
- 'brin', 'test_extensions',
- 'test_misc', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'bool_plperl', 'commit_ts',
+ 'hstore_plperl', 'hstore_plpython',
+ 'intagg', 'jsonb_plperl',
+ 'jsonb_plpython', 'ltree_plpython',
+ 'pgcrypto', 'sepgsql',
+ 'brin', 'test_extensions',
+ 'test_pg_dump', 'snapshot_too_old',
+ 'unsafe_tests', 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
--
2.20.1
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2020-07-14 19:18 ` Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Andres Freund @ 2020-07-14 19:18 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Nikhil Sontakke <nikhils@2ndquadrant.com>; Dmitry Dolgov <9erthalion6@gmail.com>; vaishnaviprabakaran@gmail.com, Craig Ringer <craig@2ndquadrant.com>; daniel@manitou-mail.org, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; Michael Paquier <michael.paquier@gmail.com>; Daniel Gustafsson <daniel@yesql.se>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; pgsql-hackers; m.kniep@web.de, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>; iwata.aya@jp.fujitsu.com, Michael Paquier <michael@paquier.xyz>
Hi,
On 2020-07-10 19:01:49 -0400, Alvaro Herrera wrote:
> Totally unasked for, here's a rebase of this patch series. I didn't do
> anything other than rebasing to current master, solving a couple of very
> trivial conflicts, fixing some whitespace complaints by git apply, and
> running tests to verify everthing works.
>
> I don't foresee working on this at all, so if anyone is interested in
> seeing this feature in, I encourage them to read and address
> Horiguchi-san's feedback.
Nor am I planning to do so, but I do think its a pretty important
improvement.
> +/*
> + * PQrecyclePipelinedCommand
> + * Push a command queue entry onto the freelist. It must be a dangling entry
> + * with null next pointer and not referenced by any other entry's next pointer.
> + */
Dangling sounds a bit like it's already freed.
> +/*
> + * PQbatchSendQueue
> + * End a batch submission by sending a protocol sync. The connection will
> + * remain in batch mode and unavailable for new synchronous command execution
> + * functions until all results from the batch are processed by the client.
I feel like the reference to the protocol sync is a bit too low level
for an external API. It should first document what the function does
from a user's POV.
I think it'd also be good to document whether / whether not queries can
already have been sent before PQbatchSendQueue is called or not.
> +/*
> + * PQbatchProcessQueue
> + * In batch mode, start processing the next query in the queue.
> + *
> + * Returns 1 if the next query was popped from the queue and can
> + * be processed by PQconsumeInput, PQgetResult, etc.
> + *
> + * Returns 0 if the current query isn't done yet, the connection
> + * is not in a batch, or there are no more queries to process.
> + */
> +int
> +PQbatchProcessQueue(PGconn *conn)
> +{
> + PGcommandQueueEntry *next_query;
> +
> + if (!conn)
> + return 0;
> +
> + if (conn->batch_status == PQBATCH_MODE_OFF)
> + return 0;
> +
> + switch (conn->asyncStatus)
> + {
> + case PGASYNC_COPY_IN:
> + case PGASYNC_COPY_OUT:
> + case PGASYNC_COPY_BOTH:
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext_noop("internal error, COPY in batch mode"));
> + break;
Shouldn't there be a return 0 here?
> + if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
> + {
> + /*
> + * In an aborted batch we don't get anything from the server for each
> + * result; we're just discarding input until we get to the next sync
> + * from the server. The client needs to know its queries got aborted
> + * so we create a fake PGresult to return immediately from
> + * PQgetResult.
> + */
> + conn->result = PQmakeEmptyPGresult(conn,
> + PGRES_BATCH_ABORTED);
> + if (!conn->result)
> + {
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext("out of memory"));
> + pqSaveErrorResult(conn);
> + return 0;
Is there any way an application can recover at this point? ISTM we'd be
stuck in the previous asyncStatus, no?
> +/* pqBatchFlush
> + * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
> + * In non-batch mode, data will be flushed all the time.
> + */
> +static int
> +pqBatchFlush(PGconn *conn)
> +{
> + if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
> + return(pqFlush(conn));
> + return 0; /* Just to keep compiler quiet */
> +}
unnecessarily long line.
> +/*
> + * Connection's outbuffer threshold is set to 64k as it is safe
> + * in Windows as per comments in pqSendSome() API.
> + */
> +#define OUTBUFFER_THRESHOLD 65536
I don't think the comment explains much. It's fine to send more than 64k
with pqSendSome(), they'll just be send with separate pgsecure_write()
invocations. And only on windows.
It clearly makes sense to start sending out data at a certain
granularity to avoid needing unnecessary amounts of memory, and to make
more efficient use of latency / serer side compute.
It's not implausible that 64k is the right amount for that, I just don't
think the explanation above is good.
> diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
> new file mode 100644
> index 0000000000..4d6ba266e5
> --- /dev/null
> +++ b/src/test/modules/test_libpq/testlibpqbatch.c
> @@ -0,0 +1,1456 @@
> +/*
> + * src/test/modules/test_libpq/testlibpqbatch.c
> + *
> + *
> + * testlibpqbatch.c
> + * Test of batch execution functionality
> + */
> +
> +#ifdef WIN32
> +#include <windows.h>
> +#endif
ISTM that this shouldn't be needed in a test program like this?
Shouldn't libpq abstract all of this away?
> +static void
> +simple_batch(PGconn *conn)
> +{
ISTM that all or at least several of these should include tests of
transactional behaviour with pipelining (e.g. using both implicit and
explicit transactions inside a single batch, using transactions across
batches, multiple explicit transactions inside a batch).
> + PGresult *res = NULL;
> + const char *dummy_params[1] = {"1"};
> + Oid dummy_param_oids[1] = {INT4OID};
> +
> + fprintf(stderr, "simple batch... ");
> + fflush(stderr);
Why do we need fflush()? IMO that shouldn't be needed in a use like
this? Especially not on stderr, which ought to be unbuffered?
> + /*
> + * Enter batch mode and dispatch a set of operations, which we'll then
> + * process the results of as they come in.
> + *
> + * For a simple case we should be able to do this without interim
> + * processing of results since our out buffer will give us enough slush to
> + * work with and we won't block on sending. So blocking mode is fine.
> + */
> + if (PQisnonblocking(conn))
> + {
> + fprintf(stderr, "Expected blocking connection mode\n");
> + goto fail;
> + }
Perhaps worth adding a helper for this?
#define EXPECT(condition, explanation) \
...
or such?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2021-03-04 11:22 ` Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Ibrar Ahmed @ 2021-03-04 11:22 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Nikhil Sontakke <nikhils@2ndquadrant.com>; Dmitry Dolgov <9erthalion6@gmail.com>; vaishnaviprabakaran@gmail.com, Craig Ringer <craig@2ndquadrant.com>; daniel@manitou-mail.org, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; Michael Paquier <michael.paquier@gmail.com>; Daniel Gustafsson <daniel@yesql.se>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; pgsql-hackers; m.kniep@web.de, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>; iwata.aya@jp.fujitsu.com, Michael Paquier <michael@paquier.xyz>
On Wed, Jul 15, 2020 at 12:18 AM Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> On 2020-07-10 19:01:49 -0400, Alvaro Herrera wrote:
> > Totally unasked for, here's a rebase of this patch series. I didn't do
> > anything other than rebasing to current master, solving a couple of very
> > trivial conflicts, fixing some whitespace complaints by git apply, and
> > running tests to verify everthing works.
> >
> > I don't foresee working on this at all, so if anyone is interested in
> > seeing this feature in, I encourage them to read and address
> > Horiguchi-san's feedback.
>
> Nor am I planning to do so, but I do think its a pretty important
> improvement.
>
>
>
> > +/*
> > + * PQrecyclePipelinedCommand
> > + * Push a command queue entry onto the freelist. It must be a
> dangling entry
> > + * with null next pointer and not referenced by any other entry's
> next pointer.
> > + */
>
> Dangling sounds a bit like it's already freed.
>
>
>
> > +/*
> > + * PQbatchSendQueue
> > + * End a batch submission by sending a protocol sync. The connection
> will
> > + * remain in batch mode and unavailable for new synchronous command
> execution
> > + * functions until all results from the batch are processed by the
> client.
>
> I feel like the reference to the protocol sync is a bit too low level
> for an external API. It should first document what the function does
> from a user's POV.
>
> I think it'd also be good to document whether / whether not queries can
> already have been sent before PQbatchSendQueue is called or not.
>
>
> > +/*
> > + * PQbatchProcessQueue
> > + * In batch mode, start processing the next query in the queue.
> > + *
> > + * Returns 1 if the next query was popped from the queue and can
> > + * be processed by PQconsumeInput, PQgetResult, etc.
> > + *
> > + * Returns 0 if the current query isn't done yet, the connection
> > + * is not in a batch, or there are no more queries to process.
> > + */
> > +int
> > +PQbatchProcessQueue(PGconn *conn)
> > +{
> > + PGcommandQueueEntry *next_query;
> > +
> > + if (!conn)
> > + return 0;
> > +
> > + if (conn->batch_status == PQBATCH_MODE_OFF)
> > + return 0;
> > +
> > + switch (conn->asyncStatus)
> > + {
> > + case PGASYNC_COPY_IN:
> > + case PGASYNC_COPY_OUT:
> > + case PGASYNC_COPY_BOTH:
> > + printfPQExpBuffer(&conn->errorMessage,
> > + libpq_gettext_noop("internal error,
> COPY in batch mode"));
> > + break;
>
> Shouldn't there be a return 0 here?
>
>
>
> > + if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass
> != PGQUERY_SYNC)
> > + {
> > + /*
> > + * In an aborted batch we don't get anything from the
> server for each
> > + * result; we're just discarding input until we get to the
> next sync
> > + * from the server. The client needs to know its queries
> got aborted
> > + * so we create a fake PGresult to return immediately from
> > + * PQgetResult.
> > + */
> > + conn->result = PQmakeEmptyPGresult(conn,
> > +
> PGRES_BATCH_ABORTED);
> > + if (!conn->result)
> > + {
> > + printfPQExpBuffer(&conn->errorMessage,
> > +
> libpq_gettext("out of memory"));
> > + pqSaveErrorResult(conn);
> > + return 0;
>
> Is there any way an application can recover at this point? ISTM we'd be
> stuck in the previous asyncStatus, no?
>
>
> > +/* pqBatchFlush
> > + * In batch mode, data will be flushed only when the out buffer reaches
> the threshold value.
> > + * In non-batch mode, data will be flushed all the time.
> > + */
> > +static int
> > +pqBatchFlush(PGconn *conn)
> > +{
> > + if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >=
> OUTBUFFER_THRESHOLD))
> > + return(pqFlush(conn));
> > + return 0; /* Just to keep compiler quiet */
> > +}
>
> unnecessarily long line.
>
>
> > +/*
> > + * Connection's outbuffer threshold is set to 64k as it is safe
> > + * in Windows as per comments in pqSendSome() API.
> > + */
> > +#define OUTBUFFER_THRESHOLD 65536
>
> I don't think the comment explains much. It's fine to send more than 64k
> with pqSendSome(), they'll just be send with separate pgsecure_write()
> invocations. And only on windows.
>
> It clearly makes sense to start sending out data at a certain
> granularity to avoid needing unnecessary amounts of memory, and to make
> more efficient use of latency / serer side compute.
>
> It's not implausible that 64k is the right amount for that, I just don't
> think the explanation above is good.
>
> > diff --git a/src/test/modules/test_libpq/testlibpqbatch.c
> b/src/test/modules/test_libpq/testlibpqbatch.c
> > new file mode 100644
> > index 0000000000..4d6ba266e5
> > --- /dev/null
> > +++ b/src/test/modules/test_libpq/testlibpqbatch.c
> > @@ -0,0 +1,1456 @@
> > +/*
> > + * src/test/modules/test_libpq/testlibpqbatch.c
> > + *
> > + *
> > + * testlibpqbatch.c
> > + * Test of batch execution functionality
> > + */
> > +
> > +#ifdef WIN32
> > +#include <windows.h>
> > +#endif
>
> ISTM that this shouldn't be needed in a test program like this?
> Shouldn't libpq abstract all of this away?
>
>
> > +static void
> > +simple_batch(PGconn *conn)
> > +{
>
> ISTM that all or at least several of these should include tests of
> transactional behaviour with pipelining (e.g. using both implicit and
> explicit transactions inside a single batch, using transactions across
> batches, multiple explicit transactions inside a batch).
>
>
>
> > + PGresult *res = NULL;
> > + const char *dummy_params[1] = {"1"};
> > + Oid dummy_param_oids[1] = {INT4OID};
> > +
> > + fprintf(stderr, "simple batch... ");
> > + fflush(stderr);
>
> Why do we need fflush()? IMO that shouldn't be needed in a use like
> this? Especially not on stderr, which ought to be unbuffered?
>
>
> > + /*
> > + * Enter batch mode and dispatch a set of operations, which we'll
> then
> > + * process the results of as they come in.
> > + *
> > + * For a simple case we should be able to do this without interim
> > + * processing of results since our out buffer will give us enough
> slush to
> > + * work with and we won't block on sending. So blocking mode is
> fine.
> > + */
> > + if (PQisnonblocking(conn))
> > + {
> > + fprintf(stderr, "Expected blocking connection mode\n");
> > + goto fail;
> > + }
>
> Perhaps worth adding a helper for this?
>
> #define EXPECT(condition, explanation) \
> ...
> or such?
>
> Greetings,
>
> Andres Freund
>
>
>
The build is failing for this patch, can you please take a look at this?
https://cirrus-ci.com/task/4568547922804736
https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.129221
I am marking the patch "Waiting on Author"
--
Ibrar Ahmed
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
@ 2021-03-04 14:26 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-04 14:26 UTC (permalink / raw)
To: PostgreSQL-development <pgsql-hackers@lists.postgresql.org>; +Cc: Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Justin Pryzby <pryzby@telsasoft.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>
On 2021-Mar-04, Ibrar Ahmed wrote:
> The build is failing for this patch, can you please take a look at this?
>
> https://cirrus-ci.com/task/4568547922804736
> https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.129221
>
>
> I am marking the patch "Waiting on Author"
I don't know why you chose to respond to such an old message, but here's
a rebase which also includes the workaround I suggested last night for
the problem of the outdated query printed by error messages, as well as
Justin's suggested fixes. (I also made a few tweaks to the TAP test).
--
Álvaro Herrera Valdivia, Chile
"No renuncies a nada. No te aferres a nada."
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-04 15:01 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:41 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 2 replies; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-04 15:01 UTC (permalink / raw)
To: PostgreSQL-development <pgsql-hackers@lists.postgresql.org>; +Cc: Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Justin Pryzby <pryzby@telsasoft.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
Apparently, the archives system or the commitfest system is not picking
up new messages to the thread, so the CF app is trying to apply a
very old patch version. I'm not sure what's up with that. Thomas, any
clues on where to look?
--
Álvaro Herrera Valdivia, Chile
"Oh, great altar of passive entertainment, bestow upon me thy discordant images
at such speed as to render linear thought impossible" (Calvin a la TV)
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-04 16:41 ` Ibrar Ahmed <ibrar.ahmad@gmail.com>
1 sibling, 0 replies; 182+ messages in thread
From: Ibrar Ahmed @ 2021-03-04 16:41 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: PostgreSQL-development <pgsql-hackers@lists.postgresql.org>; Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Justin Pryzby <pryzby@telsasoft.com>; Thomas Munro <thomas.munro@gmail.com>
On Thu, Mar 4, 2021 at 8:01 PM Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:
> Apparently, the archives system or the commitfest system is not picking
> up new messages to the thread, so the CF app is trying to apply a
> very old patch version. I'm not sure what's up with that. Thomas, any
> clues on where to look?
>
> --
> Álvaro Herrera Valdivia, Chile
> "Oh, great altar of passive entertainment, bestow upon me thy discordant
> images
> at such speed as to render linear thought impossible" (Calvin a la TV)
>
Hi Alvaro,
The thread splits and CF app still has the previous thread. The old thread
has the v18 as the latest patch which is failing. The new thread which (
https://www.postgresql.org/message-id/CAJkzx4T5E-2cQe3dtv2R78dYFvz%2Bin8PY7A8MArvLhs_pg75gg%40mail.g...
)
has the new patch.
--
Ibrar Ahmed
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-04 16:47 ` Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 1 reply; 182+ messages in thread
From: Justin Pryzby @ 2021-03-04 16:47 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
On Thu, Mar 04, 2021 at 12:01:37PM -0300, Alvaro Herrera wrote:
> Apparently, the archives system or the commitfest system is not picking
> up new messages to the thread, so the CF app is trying to apply a
> very old patch version. I'm not sure what's up with that. Thomas, any
> clues on where to look?
I think it's just because you forgot the patch.
https://www.postgresql.org/message-id/20210304142627.GA5978%40alvherre.pgsql
--
Justin
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
@ 2021-03-04 17:05 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-04 17:05 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
> I think it's just because you forgot the patch.
> https://www.postgresql.org/message-id/20210304142627.GA5978%40alvherre.pgsql
--
Álvaro Herrera 39°49'30"S 73°17'W
Attachments:
v29-libpq-pipeline.patch (text/x-diff, 103.9K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0553279314..c16befa314 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3173,6 +3173,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4919,6 +4946,498 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch a connection into pipeline mode.
+ Enter pipeline mode with <xref linkend="libpq-PQenterPipelineMode"/>
+ or test whether pipeline mode is active with
+ <xref linkend="libpq-PQpipelineStatus"/>.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using any synchronous command execution functions,
+ such as <function>PQfn</function>, or <function>PQexec</function>
+ and its sibling functions, is an error condition.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ except <literal>PGRES_PIPELINE_SYNC</literal> and null;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a pipeline causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the pipeline
+ synchronization message. The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function>, will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/test_libpq/pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-end">
+ <title>Ending Pipeline Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term>
+ <function>PQpipelineStatus</function>
+ <indexterm>
+ <primary>PQpipelineStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term>
+ <function>PQenterPipelineMode</function>
+ <indexterm>
+ <primary>PQenterPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term>
+ <function>PQexitPipelineMode</function>
+ <indexterm>
+ <primary>PQexitPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term>
+ <function>PQsendPipeline</function>
+ <indexterm>
+ <primary>PQsendPipeline</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4959,6 +5478,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2ec0580a79..75448162ca 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1109,6 +1109,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1269,6 +1275,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31a4df45f5..fbbe270654 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -484,7 +485,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2504,6 +2507,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2693,11 +2700,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2741,8 +2762,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2810,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3066,13 +3095,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3212,7 +3264,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3456,6 +3516,49 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4683,7 +4786,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6808,4 +6913,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index f83af03d0a..4c97e8a078 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -540,6 +540,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -571,6 +588,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2486,6 +2509,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3928,6 +3952,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4187,6 +4212,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6735,6 +6761,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0fb790fdfa 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1178,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1197,18 +1204,37 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQuery");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, true);
}
int
PQsendQueryContinue(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQueryContinue");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, false);
}
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ /* pipeline mode requires extended query protocol */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
@@ -1307,6 +1333,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1358,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1393,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1480,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1502,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1576,34 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+ /*
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1713,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1726,14 +1847,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1927,49 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-related result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server.
+ *
+ * In all other cases, leave the queue state change for next
+ * time, so that a terminating NULL result is sent.
+ */
+ if (res && (res->resultStatus == PGRES_PIPELINE_ABORTED ||
+ res->resultStatus == PGRES_PIPELINE_SYNC))
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2150,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2320,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2330,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2350,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2541,6 +2736,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2757,359 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3152,6 +3707,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..b131995974 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're also IDLE when in pipeline mode we have completed
+ * processing the results of one query and are waiting for the
+ * next one in the pipeline. In this case, as above, just wait to
+ * see what's next.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -450,7 +478,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -875,6 +903,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -930,9 +962,21 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* Save the active query text, if any, into res as well; but only if we
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
+ *
+ * Note that in pipeline mode, we have not yet advanced the query pointer
+ * to the next query, so we have to look at that.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res)
+ {
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->asyncStatus == PGASYNC_IDLE)
+ {
+ if (conn->cmd_queue_head)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
+ }
+ else if (conn->last_query)
+ res->errQuery = pqResultStrdup(res, conn->last_query);
+ }
/*
* Now build the "overall" error message for PQresultErrorMessage.
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 8d51e6ed9f..3e9f99f8fb 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -397,6 +418,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -409,6 +431,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -790,6 +822,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..6e31458be2 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..4fbf97a5b0
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/pipeline
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..64a3a4af5e
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/test_libpq/Makefile
+
+PROGRAM = pipeline
+OBJS = pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/pipeline.c b/src/test/modules/test_libpq/pipeline.c
new file mode 100644
index 0000000000..bb217e6b08
--- /dev/null
+++ b/src/test/modules/test_libpq/pipeline.c
@@ -0,0 +1,1141 @@
+/*
+ * src/test/modules/test_libpq/pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQueryParams(conn, "SELECT 1/g FROM generate_series(5, -1, -1) g",
+ 0, NULL, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ if (PQresultStatus(res) == PGRES_FATAL_ERROR)
+ printf("got error: {{%s}} (expected: division by zero)\n", PQerrorMessage(conn));
+ else if (PQresultStatus(res) == PGRES_SINGLE_TUPLE)
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ else
+ pg_fatal("got unexpected result %s\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ }
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "%d: got NULL result\n", i);
+ continue;
+ }
+ restype = PQresultStatus(res);
+ fprintf(stderr, "%d: got status %s", i, PQresStatus(restype));
+ if (restype == PGRES_FATAL_ERROR)
+ fprintf(stderr, "; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ fprintf(stderr, ": command didn't run because pipeline aborted\n");
+ }
+ else
+ fprintf(stderr, "\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple\n");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple\n");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..c7d4b1feec
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8bd95aefa1..1bd23eec6d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-04 19:24 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-04 19:24 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
v30 contains changes to hopefully make it build on MSVC.
--
Álvaro Herrera Valdivia, Chile
Attachments:
v30-libpq-pipeline.patch (text/x-diff, 104.6K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0553279314..c16befa314 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3173,6 +3173,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4919,6 +4946,498 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch a connection into pipeline mode.
+ Enter pipeline mode with <xref linkend="libpq-PQenterPipelineMode"/>
+ or test whether pipeline mode is active with
+ <xref linkend="libpq-PQpipelineStatus"/>.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using any synchronous command execution functions,
+ such as <function>PQfn</function>, or <function>PQexec</function>
+ and its sibling functions, is an error condition.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ except <literal>PGRES_PIPELINE_SYNC</literal> and null;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a pipeline causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the pipeline
+ synchronization message. The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function>, will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/test_libpq/pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-end">
+ <title>Ending Pipeline Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term>
+ <function>PQpipelineStatus</function>
+ <indexterm>
+ <primary>PQpipelineStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term>
+ <function>PQenterPipelineMode</function>
+ <indexterm>
+ <primary>PQenterPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term>
+ <function>PQexitPipelineMode</function>
+ <indexterm>
+ <primary>PQexitPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term>
+ <function>PQsendPipeline</function>
+ <indexterm>
+ <primary>PQsendPipeline</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4959,6 +5478,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2ec0580a79..75448162ca 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1109,6 +1109,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1269,6 +1275,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31a4df45f5..fbbe270654 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -484,7 +485,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2504,6 +2507,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2693,11 +2700,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2741,8 +2762,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2810,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3066,13 +3095,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3212,7 +3264,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3456,6 +3516,49 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4683,7 +4786,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6808,4 +6913,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index f83af03d0a..4c97e8a078 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -540,6 +540,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -571,6 +588,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2486,6 +2509,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3928,6 +3952,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4187,6 +4212,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6735,6 +6761,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0fb790fdfa 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1178,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1197,18 +1204,37 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQuery");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, true);
}
int
PQsendQueryContinue(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQueryContinue");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, false);
}
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ /* pipeline mode requires extended query protocol */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
@@ -1307,6 +1333,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1358,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1393,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1480,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1502,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1576,34 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+ /*
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1713,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1726,14 +1847,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1927,49 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-related result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server.
+ *
+ * In all other cases, leave the queue state change for next
+ * time, so that a terminating NULL result is sent.
+ */
+ if (res && (res->resultStatus == PGRES_PIPELINE_ABORTED ||
+ res->resultStatus == PGRES_PIPELINE_SYNC))
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2150,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2320,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2330,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2350,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2541,6 +2736,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2757,359 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3152,6 +3707,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..b131995974 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're also IDLE when in pipeline mode we have completed
+ * processing the results of one query and are waiting for the
+ * next one in the pipeline. In this case, as above, just wait to
+ * see what's next.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -450,7 +478,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -875,6 +903,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -930,9 +962,21 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* Save the active query text, if any, into res as well; but only if we
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
+ *
+ * Note that in pipeline mode, we have not yet advanced the query pointer
+ * to the next query, so we have to look at that.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res)
+ {
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->asyncStatus == PGASYNC_IDLE)
+ {
+ if (conn->cmd_queue_head)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
+ }
+ else if (conn->last_query)
+ res->errQuery = pqResultStrdup(res, conn->last_query);
+ }
/*
* Now build the "overall" error message for PQresultErrorMessage.
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 8d51e6ed9f..3e9f99f8fb 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -397,6 +418,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -409,6 +431,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -790,6 +822,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..6e31458be2 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..4fbf97a5b0
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/pipeline
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..2b00877c93
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/test_libpq/Makefile
+
+PROGRAM = pipeline
+OBJS = pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/pipeline.c b/src/test/modules/test_libpq/pipeline.c
new file mode 100644
index 0000000000..bb217e6b08
--- /dev/null
+++ b/src/test/modules/test_libpq/pipeline.c
@@ -0,0 +1,1141 @@
+/*
+ * src/test/modules/test_libpq/pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQueryParams(conn, "SELECT 1/g FROM generate_series(5, -1, -1) g",
+ 0, NULL, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ if (PQresultStatus(res) == PGRES_FATAL_ERROR)
+ printf("got error: {{%s}} (expected: division by zero)\n", PQerrorMessage(conn));
+ else if (PQresultStatus(res) == PGRES_SINGLE_TUPLE)
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ else
+ pg_fatal("got unexpected result %s\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ }
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "%d: got NULL result\n", i);
+ continue;
+ }
+ restype = PQresultStatus(res);
+ fprintf(stderr, "%d: got status %s", i, PQresStatus(restype));
+ if (restype == PGRES_FATAL_ERROR)
+ fprintf(stderr, "; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ fprintf(stderr, ": command didn't run because pipeline aborted\n");
+ }
+ else
+ fprintf(stderr, "\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple\n");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple\n");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..c7d4b1feec
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 49614106dc..b2546f0809 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,7 +33,7 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
+my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'test_libpq');
my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
my $contrib_extralibs = undef;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8bd95aefa1..1bd23eec6d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-04 19:48 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-04 19:48 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
On 2021-Mar-04, Alvaro Herrera wrote:
> v30 contains changes to hopefully make it build on MSVC.
Hm, that didn't work -- appveyor still says:
Project "C:\projects\postgresql\pgsql.sln" (1) is building "C:\projects\postgresql\pipeline.vcxproj" (75) on node 1 (default targets).
PrepareForBuild:
Creating directory ".\Release\pipeline\".
Creating directory ".\Release\pipeline\pipeline.tlog\".
InitializeBuildStatus:
Creating ".\Release\pipeline\pipeline.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
ClCompile:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\x86_amd64\CL.exe /c /Isrc/include /Isrc/include/port/win32 /Isrc/include/port/win32_msvc /Zi /nologo /W3 /WX- /Ox /D WIN32 /D _WINDOWS /D __WINDOWS__ /D __WIN32__ /D WIN32_STACK_RLIMIT=4194304 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _MBCS /GF /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t /Zc:forScope /Fo".\Release\pipeline\\" /Fd".\Release\pipeline\vc120.pdb" /Gd /TC /wd4018 /wd4244 /wd4273 /wd4102 /wd4090 /wd4267 /errorReport:queue /MP src/test/modules/test_libpq/pipeline.c
pipeline.c
src/test/modules/test_libpq/pipeline.c(11): fatal error C1083: Cannot open include file: 'libpq-fe.h': No such file or directory [C:\projects\postgresql\pipeline.vcxproj]
Done Building Project "C:\projects\postgresql\pipeline.vcxproj" (default targets) -- FAILED.
Project "C:\projects\postgresql\pgsql.sln" (1) is building "C:\projects\postgresql\test_parser.vcxproj" (76) on node 1 (default targets).
I think the problem is that the project is called pipeline and not test_libpq,
so there's no match in the name. I'm going to rename the whole thing to
src/test/modules/libpq_pipeline/ and see if the msvc tooling likes that
better.
--
Álvaro Herrera 39°49'30"S 73°17'W
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-04 19:59 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 20:51 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-04 19:59 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
On 2021-Mar-04, Alvaro Herrera wrote:
> I think the problem is that the project is called pipeline and not
> test_libpq, so there's no match in the name. I'm going to rename the
> whole thing to src/test/modules/libpq_pipeline/ and see if the msvc
> tooling likes that better.
v31.
--
Álvaro Herrera Valdivia, Chile
"I'm impressed how quickly you are fixing this obscure issue. I came from
MS SQL and it would be hard for me to put into words how much of a better job
you all are doing on [PostgreSQL]."
Steve Midgley, http://archives.postgresql.org/pgsql-sql/2008-08/msg00000.php
Attachments:
v31-libpq-pipeline.patch (text/x-diff, 104.7K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0553279314..c87b0ce911 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3173,6 +3173,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4919,6 +4946,498 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch a connection into pipeline mode.
+ Enter pipeline mode with <xref linkend="libpq-PQenterPipelineMode"/>
+ or test whether pipeline mode is active with
+ <xref linkend="libpq-PQpipelineStatus"/>.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using any synchronous command execution functions,
+ such as <function>PQfn</function>, or <function>PQexec</function>
+ and its sibling functions, is an error condition.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ except <literal>PGRES_PIPELINE_SYNC</literal> and null;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a pipeline causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the pipeline
+ synchronization message. The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function>, will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-end">
+ <title>Ending Pipeline Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term>
+ <function>PQpipelineStatus</function>
+ <indexterm>
+ <primary>PQpipelineStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term>
+ <function>PQenterPipelineMode</function>
+ <indexterm>
+ <primary>PQenterPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term>
+ <function>PQexitPipelineMode</function>
+ <indexterm>
+ <primary>PQexitPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term>
+ <function>PQsendPipeline</function>
+ <indexterm>
+ <primary>PQsendPipeline</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4959,6 +5478,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2ec0580a79..75448162ca 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1109,6 +1109,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1269,6 +1275,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31a4df45f5..fbbe270654 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -484,7 +485,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2504,6 +2507,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2693,11 +2700,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2741,8 +2762,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2810,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3066,13 +3095,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3212,7 +3264,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3456,6 +3516,49 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4683,7 +4786,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6808,4 +6913,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index f83af03d0a..4c97e8a078 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -540,6 +540,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -571,6 +588,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2486,6 +2509,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3928,6 +3952,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4187,6 +4212,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6735,6 +6761,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0fb790fdfa 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1178,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1197,18 +1204,37 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQuery");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, true);
}
int
PQsendQueryContinue(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQueryContinue");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, false);
}
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ /* pipeline mode requires extended query protocol */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
@@ -1307,6 +1333,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1358,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1393,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1480,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1502,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1576,34 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+ /*
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1713,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1726,14 +1847,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1927,49 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-related result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server.
+ *
+ * In all other cases, leave the queue state change for next
+ * time, so that a terminating NULL result is sent.
+ */
+ if (res && (res->resultStatus == PGRES_PIPELINE_ABORTED ||
+ res->resultStatus == PGRES_PIPELINE_SYNC))
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2150,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2320,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2330,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2350,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2541,6 +2736,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2757,359 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3152,6 +3707,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..b131995974 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're also IDLE when in pipeline mode we have completed
+ * processing the results of one query and are waiting for the
+ * next one in the pipeline. In this case, as above, just wait to
+ * see what's next.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -450,7 +478,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -875,6 +903,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -930,9 +962,21 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* Save the active query text, if any, into res as well; but only if we
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
+ *
+ * Note that in pipeline mode, we have not yet advanced the query pointer
+ * to the next query, so we have to look at that.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res)
+ {
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->asyncStatus == PGASYNC_IDLE)
+ {
+ if (conn->cmd_queue_head)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
+ }
+ else if (conn->last_query)
+ res->errQuery = pqResultStrdup(res, conn->last_query);
+ }
/*
* Now build the "overall" error message for PQresultErrorMessage.
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 8d51e6ed9f..3e9f99f8fb 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -397,6 +418,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -409,6 +431,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -790,6 +822,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..41706b514c
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1141 @@
+/*
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQueryParams(conn, "SELECT 1/g FROM generate_series(5, -1, -1) g",
+ 0, NULL, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ if (PQresultStatus(res) == PGRES_FATAL_ERROR)
+ printf("got error: {{%s}} (expected: division by zero)\n", PQerrorMessage(conn));
+ else if (PQresultStatus(res) == PGRES_SINGLE_TUPLE)
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ else
+ pg_fatal("got unexpected result %s\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ }
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "%d: got NULL result\n", i);
+ continue;
+ }
+ restype = PQresultStatus(res);
+ fprintf(stderr, "%d: got status %s", i, PQresStatus(restype));
+ if (restype == PGRES_FATAL_ERROR)
+ fprintf(stderr, "; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ fprintf(stderr, ": command didn't run because pipeline aborted\n");
+ }
+ else
+ fprintf(stderr, "\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple\n");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple\n");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..a12f2dd47b
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libp_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 49614106dc..ed65a61a62 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,7 +33,8 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
my $contrib_extralibs = undef;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8bd95aefa1..1bd23eec6d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-04 20:51 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 21:40 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-04 20:51 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
On 2021-Mar-04, Alvaro Herrera wrote:
> v31.
Got this:
libpq_pipeline.obj : error LNK2019: unresolved external symbol __WSAFDIsSet referenced in function test_pipelined_insert [C:\projects\postgresql\libpq_pipeline.vcxproj]
5019libpq_pipeline.obj : error LNK2019: unresolved external symbol __imp_select referenced in function test_pipelined_insert [C:\projects\postgresql\libpq_pipeline.vcxproj]
5020libpq_pipeline.obj : error LNK2019: unresolved external symbol pg_snprintf referenced in function test_pipelined_insert [C:\projects\postgresql\libpq_pipeline.vcxproj]
5021libpq_pipeline.obj : error LNK2019: unresolved external symbol pg_vfprintf referenced in function pg_fatal_impl [C:\projects\postgresql\libpq_pipeline.vcxproj]
5022libpq_pipeline.obj : error LNK2019: unresolved external symbol pg_fprintf referenced in function pg_fatal_impl [C:\projects\postgresql\libpq_pipeline.vcxproj]
5023libpq_pipeline.obj : error LNK2019: unresolved external symbol pg_printf referenced in function pg_fatal_impl [C:\projects\postgresql\libpq_pipeline.vcxproj]
5024libpq_pipeline.obj : error LNK2019: unresolved external symbol pg_strerror referenced in function test_pipelined_insert [C:\projects\postgresql\libpq_pipeline.vcxproj]
5025libpq_pipeline.obj : error LNK2019: unresolved external symbol pg_strdup referenced in function main [C:\projects\postgresql\libpq_pipeline.vcxproj]
5026libpq_pipeline.obj : error LNK2019: unresolved external symbol pfree referenced in function test_singlerowmode [C:\projects\postgresql\libpq_pipeline.vcxproj]
5027libpq_pipeline.obj : error LNK2019: unresolved external symbol psprintf referenced in function test_singlerowmode [C:\projects\postgresql\libpq_pipeline.vcxproj]
pg_snprintf, pg_vfprintf, pg_fprintf, pg_printf, pg_strerror are in pgport.
pg_strdup and pfree, psprintf are in pgcommon.
I don't know where do __WSAFDIsSet and __imp_select come from or what to
do about them. Let's see if adding pgport and pgcommon fixes things.
--
Álvaro Herrera 39°49'30"S 73°17'W
Attachments:
v32-libpq-pipeline.patch (text/x-diff, 104.9K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0553279314..c87b0ce911 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3173,6 +3173,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4919,6 +4946,498 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch a connection into pipeline mode.
+ Enter pipeline mode with <xref linkend="libpq-PQenterPipelineMode"/>
+ or test whether pipeline mode is active with
+ <xref linkend="libpq-PQpipelineStatus"/>.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using any synchronous command execution functions,
+ such as <function>PQfn</function>, or <function>PQexec</function>
+ and its sibling functions, is an error condition.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ except <literal>PGRES_PIPELINE_SYNC</literal> and null;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a pipeline causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the pipeline
+ synchronization message. The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function>, will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-end">
+ <title>Ending Pipeline Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term>
+ <function>PQpipelineStatus</function>
+ <indexterm>
+ <primary>PQpipelineStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term>
+ <function>PQenterPipelineMode</function>
+ <indexterm>
+ <primary>PQenterPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term>
+ <function>PQexitPipelineMode</function>
+ <indexterm>
+ <primary>PQexitPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term>
+ <function>PQsendPipeline</function>
+ <indexterm>
+ <primary>PQsendPipeline</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4959,6 +5478,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2ec0580a79..75448162ca 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1109,6 +1109,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1269,6 +1275,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31a4df45f5..fbbe270654 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -484,7 +485,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2504,6 +2507,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2693,11 +2700,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2741,8 +2762,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2810,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3066,13 +3095,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3212,7 +3264,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3456,6 +3516,49 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4683,7 +4786,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6808,4 +6913,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index f83af03d0a..4c97e8a078 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -540,6 +540,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -571,6 +588,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2486,6 +2509,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3928,6 +3952,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4187,6 +4212,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6735,6 +6761,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0fb790fdfa 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1178,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1197,18 +1204,37 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQuery");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, true);
}
int
PQsendQueryContinue(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQueryContinue");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, false);
}
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ /* pipeline mode requires extended query protocol */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
@@ -1307,6 +1333,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1358,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1393,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1480,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1502,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1576,34 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+ /*
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1713,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1726,14 +1847,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1927,49 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-related result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server.
+ *
+ * In all other cases, leave the queue state change for next
+ * time, so that a terminating NULL result is sent.
+ */
+ if (res && (res->resultStatus == PGRES_PIPELINE_ABORTED ||
+ res->resultStatus == PGRES_PIPELINE_SYNC))
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2150,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2320,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2330,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2350,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2541,6 +2736,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2757,359 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3152,6 +3707,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..b131995974 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're also IDLE when in pipeline mode we have completed
+ * processing the results of one query and are waiting for the
+ * next one in the pipeline. In this case, as above, just wait to
+ * see what's next.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -450,7 +478,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -875,6 +903,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -930,9 +962,21 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* Save the active query text, if any, into res as well; but only if we
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
+ *
+ * Note that in pipeline mode, we have not yet advanced the query pointer
+ * to the next query, so we have to look at that.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res)
+ {
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->asyncStatus == PGASYNC_IDLE)
+ {
+ if (conn->cmd_queue_head)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
+ }
+ else if (conn->last_query)
+ res->errQuery = pqResultStrdup(res, conn->last_query);
+ }
/*
* Now build the "overall" error message for PQresultErrorMessage.
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 8d51e6ed9f..3e9f99f8fb 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -397,6 +418,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -409,6 +431,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -790,6 +822,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..41706b514c
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1141 @@
+/*
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQueryParams(conn, "SELECT 1/g FROM generate_series(5, -1, -1) g",
+ 0, NULL, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ if (PQresultStatus(res) == PGRES_FATAL_ERROR)
+ printf("got error: {{%s}} (expected: division by zero)\n", PQerrorMessage(conn));
+ else if (PQresultStatus(res) == PGRES_SINGLE_TUPLE)
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ else
+ pg_fatal("got unexpected result %s\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ }
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "%d: got NULL result\n", i);
+ continue;
+ }
+ restype = PQresultStatus(res);
+ fprintf(stderr, "%d: got status %s", i, PQresStatus(restype));
+ if (restype == PGRES_FATAL_ERROR)
+ fprintf(stderr, "; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ fprintf(stderr, ": command didn't run because pipeline aborted\n");
+ }
+ else
+ fprintf(stderr, "\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple\n");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple\n");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..a12f2dd47b
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libp_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 49614106dc..e3b14af9dd 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,9 +33,10 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
-my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
-my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
+my @contrib_uselibpgport = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my @contrib_uselibpgcommon = ('libpq_pipeline', 'oid2name', 'vacuumlo');
my $contrib_extralibs = undef;
my $contrib_extraincludes = { 'dblink' => ['src/backend'] };
my $contrib_extrasource = {
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8bd95aefa1..1bd23eec6d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 20:51 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-04 21:40 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-06 00:31 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Zhihong Yu <zyu@yugabyte.com>
2021-03-06 00:35 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 2 replies; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-04 21:40 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
On 2021-Mar-04, Alvaro Herrera wrote:
> I don't know where do __WSAFDIsSet and __imp_select come from or what to
> do about them. Let's see if adding pgport and pgcommon fixes things.
Indeed all those other problems were fixed and these remain. New
failure is:
"C:\projects\postgresql\pgsql.sln" (default target) (1) ->
6007"C:\projects\postgresql\libpq_pipeline.vcxproj" (default target) (55) ->
6008(Link target) ->
6009 libpq_pipeline.obj : error LNK2019: unresolved external symbol __WSAFDIsSet referenced in function test_pipelined_insert [C:\projects\postgresql\libpq_pipeline.vcxproj]
6010 libpq_pipeline.obj : error LNK2019: unresolved external symbol __imp_select referenced in function test_pipelined_insert [C:\projects\postgresql\libpq_pipeline.vcxproj]
6011 .\Release\libpq_pipeline\libpq_pipeline.exe : fatal error LNK1120: 2 unresolved externals [C:\projects\postgresql\libpq_pipeline.vcxproj]
I did notice that isolationtester.c is using select(), and one
difference is that it includes <sys/select.h> which libpq_pipeline.c
does not -- and it also pulls in ws2_32.lib. Let's see if those two
changes fix things.
--
Álvaro Herrera Valdivia, Chile
Attachments:
v33-libpq-pipeline.patch (text/x-diff, 105.1K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0553279314..c87b0ce911 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3173,6 +3173,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4919,6 +4946,498 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch a connection into pipeline mode.
+ Enter pipeline mode with <xref linkend="libpq-PQenterPipelineMode"/>
+ or test whether pipeline mode is active with
+ <xref linkend="libpq-PQpipelineStatus"/>.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using any synchronous command execution functions,
+ such as <function>PQfn</function>, or <function>PQexec</function>
+ and its sibling functions, is an error condition.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ except <literal>PGRES_PIPELINE_SYNC</literal> and null;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a pipeline causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the pipeline
+ synchronization message. The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function>, will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-end">
+ <title>Ending Pipeline Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term>
+ <function>PQpipelineStatus</function>
+ <indexterm>
+ <primary>PQpipelineStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term>
+ <function>PQenterPipelineMode</function>
+ <indexterm>
+ <primary>PQenterPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term>
+ <function>PQexitPipelineMode</function>
+ <indexterm>
+ <primary>PQexitPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term>
+ <function>PQsendPipeline</function>
+ <indexterm>
+ <primary>PQsendPipeline</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4959,6 +5478,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2ec0580a79..75448162ca 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1109,6 +1109,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1269,6 +1275,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31a4df45f5..fbbe270654 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -484,7 +485,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2504,6 +2507,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2693,11 +2700,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2741,8 +2762,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2810,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3066,13 +3095,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3212,7 +3264,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3456,6 +3516,49 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4683,7 +4786,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6808,4 +6913,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index f83af03d0a..4c97e8a078 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -540,6 +540,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -571,6 +588,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2486,6 +2509,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3928,6 +3952,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4187,6 +4212,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6735,6 +6761,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0fb790fdfa 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1178,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1197,18 +1204,37 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQuery");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, true);
}
int
PQsendQueryContinue(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQueryContinue");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, false);
}
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ /* pipeline mode requires extended query protocol */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
@@ -1307,6 +1333,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1358,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1393,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1480,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1502,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1576,34 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+ /*
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1713,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1726,14 +1847,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1927,49 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-related result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server.
+ *
+ * In all other cases, leave the queue state change for next
+ * time, so that a terminating NULL result is sent.
+ */
+ if (res && (res->resultStatus == PGRES_PIPELINE_ABORTED ||
+ res->resultStatus == PGRES_PIPELINE_SYNC))
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2150,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2320,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2330,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2350,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2541,6 +2736,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2757,359 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3152,6 +3707,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..b131995974 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're also IDLE when in pipeline mode we have completed
+ * processing the results of one query and are waiting for the
+ * next one in the pipeline. In this case, as above, just wait to
+ * see what's next.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -450,7 +478,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -875,6 +903,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -930,9 +962,21 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* Save the active query text, if any, into res as well; but only if we
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
+ *
+ * Note that in pipeline mode, we have not yet advanced the query pointer
+ * to the next query, so we have to look at that.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res)
+ {
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->asyncStatus == PGASYNC_IDLE)
+ {
+ if (conn->cmd_queue_head)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
+ }
+ else if (conn->last_query)
+ res->errQuery = pqResultStrdup(res, conn->last_query);
+ }
/*
* Now build the "overall" error message for PQresultErrorMessage.
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 8d51e6ed9f..3e9f99f8fb 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -397,6 +418,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -409,6 +431,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -790,6 +822,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..782e106e33
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1144 @@
+/*
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+#ifdef HAVE_SYS_SELECT_H
+#include <sys/select.h>
+#endif
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQueryParams(conn, "SELECT 1/g FROM generate_series(5, -1, -1) g",
+ 0, NULL, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ if (PQresultStatus(res) == PGRES_FATAL_ERROR)
+ printf("got error: {{%s}} (expected: division by zero)\n", PQerrorMessage(conn));
+ else if (PQresultStatus(res) == PGRES_SINGLE_TUPLE)
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ else
+ pg_fatal("got unexpected result %s\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ }
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "%d: got NULL result\n", i);
+ continue;
+ }
+ restype = PQresultStatus(res);
+ fprintf(stderr, "%d: got status %s", i, PQresStatus(restype));
+ if (restype == PGRES_FATAL_ERROR)
+ fprintf(stderr, "; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ fprintf(stderr, ": command didn't run because pipeline aborted\n");
+ }
+ else
+ fprintf(stderr, "\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple\n");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple\n");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..a12f2dd47b
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libp_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 49614106dc..44b1a43f30 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,10 +33,11 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
-my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
-my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
-my $contrib_extralibs = undef;
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
+my @contrib_uselibpgport = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my @contrib_uselibpgcommon = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my $contrib_extralibs = { 'libpq_pipeline' => ['ws2_32.lib'] };
my $contrib_extraincludes = { 'dblink' => ['src/backend'] };
my $contrib_extrasource = {
'cube' => [ 'contrib/cube/cubescan.l', 'contrib/cube/cubeparse.y' ],
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8bd95aefa1..1bd23eec6d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 20:51 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 21:40 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-06 00:31 ` Zhihong Yu <zyu@yugabyte.com>
1 sibling, 0 replies; 182+ messages in thread
From: Zhihong Yu @ 2021-03-06 00:31 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; PostgreSQL Developers <pgsql-hackers@lists.postgresql.org>; Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
Hi,
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
'used pipeline mode' -> 'used in pipeline mode'
inline patch 1: src/test/modules/libpq_pipeline/libpq_pipeline.c (+2/-0 lines) download inline-1.patch
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1144 @@
+/*
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 20:51 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 21:40 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-06 00:35 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-11 16:00 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-16 20:00 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
1 sibling, 2 replies; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-06 00:35 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>
v33 was indeed marked a pass by cfbot. However, it only *builds* the
test program, it does not *run* it. I guess we'll have to wait for the
buildfarm to tell us more.
In the meantime, I implemented PQsendQuery() as callable in pipeline
mode; it does that by using the extended-query protocol directly rather
than sending 'Q' as in non-pipeline mode. I also adjusted the docs a
little bit more. That's what you see here as v34.
I'll take the weekend to think about the issue with conn->last_query and
conn->queryclass that I mentioned yesterday; other than that detail my
feeling is that this is committable, so I'll be looking at getting this
pushed early next weeks, barring opinions from others.
--
Álvaro Herrera Valdivia, Chile
"El sabio habla porque tiene algo que decir;
el tonto, porque tiene que decir algo" (Platon).
Attachments:
v34-libpq-pipeline.patch (text/x-diff, 107.4K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 0553279314..e3cd5c377b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3173,6 +3173,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4919,6 +4946,473 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be
+ sent/received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch the connection
+ into pipeline mode,
+ which is done with <xref linkend="libpq-PQenterPipelineMode"/>.
+ <xref linkend="libpq-PQpipelineStatus"/> can be used
+ to test whether pipeline mode is active.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using synchronous command execution functions
+ such as <function>PQfn</function>,
+ <function>PQexec</function>,
+ <function>PQexecParams</function>,
+ <function>PQprepare</function>,
+ <function>PQexecPrepared</function>,
+ <function>PQdescribePrepared</function>,
+ <function>PQdescribePortal</function>,
+ is an error condition.
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQuery"/>,
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave"/>.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ until <literal>PGRES_PIPELINE_SYNC</literal>;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function> will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term><function>PQpipelineStatus</function><indexterm><primary>PQpipelineStatus</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term><function>PQenterPipelineMode</function><indexterm><primary>PQenterPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term><function>PQexitPipelineMode</function><indexterm><primary>PQexitPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term><function>PQsendPipeline</function><indexterm><primary>PQsendPipeline</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4959,6 +5453,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2ec0580a79..75448162ca 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1109,6 +1109,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1269,6 +1275,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31a4df45f5..fbbe270654 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -484,7 +485,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2504,6 +2507,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2693,11 +2700,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2741,8 +2762,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2810,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3066,13 +3095,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3212,7 +3264,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3456,6 +3516,49 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4683,7 +4786,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6808,4 +6913,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index f83af03d0a..4c97e8a078 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -540,6 +540,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -571,6 +588,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2486,6 +2509,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3928,6 +3952,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4187,6 +4212,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6735,6 +6761,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0cdba4260e 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1178,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1209,9 +1216,23 @@ PQsendQueryContinue(PGconn *conn, const char *query)
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
+ /*
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* check the argument */
if (!query)
{
@@ -1220,37 +1241,79 @@ PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
return 0;
}
- /* construct the outgoing Query message */
- if (pqPutMsgStart('Q', conn) < 0 ||
- pqPuts(query, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
+ /* Send the query message(s) */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- /* error message should be set up already */
- return 0;
+ /* construct the outgoing Query message */
+ if (pqPutMsgStart('Q', conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ {
+ /* error message should be set up already */
+ return 0;
+ }
+
+ /* remember we are using simple query protocol */
+ conn->queryclass = PGQUERY_SIMPLE;
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
}
+ else
+ {
+ /*
+ * In pipeline mode, we cannot use the simple protocol, so we send
+ * Parse, Bind, Describe Portal, Execute.
+ */
+ if (pqPutMsgStart('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('B', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('D', conn) < 0 ||
+ pqPutc('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('E', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 4, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
- /* remember we are using simple query protocol */
- conn->queryclass = PGQUERY_SIMPLE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ pipeCmd->queryclass = PGQUERY_EXTENDED;
+ pipeCmd->query = strdup(query);
+ }
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
- {
- /* error message should be set up already */
- return 0;
- }
+ if (pqPipelineFlush(conn) < 0)
+ goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
+ /* error message should be set up already */
+ return 0;
}
/*
@@ -1307,6 +1370,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1395,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1430,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1517,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1539,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1613,34 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+ /*
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1750,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1726,14 +1884,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1964,49 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-related result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server.
+ *
+ * In all other cases, leave the queue state change for next
+ * time, so that a terminating NULL result is sent.
+ */
+ if (res && (res->resultStatus == PGRES_PIPELINE_ABORTED ||
+ res->resultStatus == PGRES_PIPELINE_SYNC))
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2187,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2357,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2367,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2387,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2541,6 +2773,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2794,359 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3152,6 +3744,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..b131995974 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're also IDLE when in pipeline mode we have completed
+ * processing the results of one query and are waiting for the
+ * next one in the pipeline. In this case, as above, just wait to
+ * see what's next.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -450,7 +478,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -875,6 +903,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -930,9 +962,21 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* Save the active query text, if any, into res as well; but only if we
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
+ *
+ * Note that in pipeline mode, we have not yet advanced the query pointer
+ * to the next query, so we have to look at that.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res)
+ {
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->asyncStatus == PGASYNC_IDLE)
+ {
+ if (conn->cmd_queue_head)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
+ }
+ else if (conn->last_query)
+ res->errQuery = pqResultStrdup(res, conn->last_query);
+ }
/*
* Now build the "overall" error message for PQresultErrorMessage.
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 8d51e6ed9f..3e9f99f8fb 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -397,6 +418,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -409,6 +431,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -790,6 +822,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..1074634384
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1152 @@
+/*
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+#ifdef HAVE_SYS_SELECT_H
+#include <sys/select.h>
+#endif
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush
+ * to work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+ bool goterror;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQuery(conn, "SELECT 1.0/g FROM generate_series(3, -1, -1) g") != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ goterror = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_SINGLE_TUPLE:
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ break;
+ case PGRES_FATAL_ERROR:
+ if (strcmp(PQresultErrorField(res, PG_DIAG_SQLSTATE), "22012") != 0)
+ pg_fatal("expected division-by-zero, got: %s\n",
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ printf("got expected division-by-zero\n");
+ goterror = true;
+ break;
+ default:
+ pg_fatal("got unexpected result %s\n", PQresStatus(PQresultStatus(res)));
+ }
+ PQclear(res);
+ }
+ if (!goterror)
+ pg_fatal("did not get division-by-zero error\n");
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "%d: got NULL result\n", i);
+ continue;
+ }
+ restype = PQresultStatus(res);
+ fprintf(stderr, "%d: got status %s", i, PQresStatus(restype));
+ if (restype == PGRES_FATAL_ERROR)
+ fprintf(stderr, "; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ fprintf(stderr, ": command didn't run because pipeline aborted\n");
+ }
+ else
+ fprintf(stderr, "\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple\n");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple\n");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..a12f2dd47b
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libp_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 49614106dc..44b1a43f30 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,10 +33,11 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
-my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
-my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
-my $contrib_extralibs = undef;
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
+my @contrib_uselibpgport = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my @contrib_uselibpgcommon = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my $contrib_extralibs = { 'libpq_pipeline' => ['ws2_32.lib'] };
my $contrib_extraincludes = { 'dblink' => ['src/backend'] };
my $contrib_extrasource = {
'cube' => [ 'contrib/cube/cubescan.l', 'contrib/cube/cubeparse.y' ],
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8bd95aefa1..1bd23eec6d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 20:51 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 21:40 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-06 00:35 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-11 16:00 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-11 18:49 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-11 16:00 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>
On 2021-Mar-05, Alvaro Herrera wrote:
> I'll take the weekend to think about the issue with conn->last_query and
> conn->queryclass that I mentioned yesterday; other than that detail my
> feeling is that this is committable, so I'll be looking at getting this
> pushed early next weeks, barring opinions from others.
It took longer than I expected, but it works well now. conn->last_query
is gone; all commands, both in pipeline mode and in no-pipeline mode, go
via the command queue. This is cleaner all around; we don't have to
have the pipeline code "cheat" so that it looks like each command is
"last" at each point.
I have not absorbed David Johnston's latest doc suggestions yet.
I'm going to give the code a last renaming pass, on the idea that the
command queue is no longer exclusively for the pipeline mode, so some
things need less exclusionary names. But functionality wise AFAICS this
patch has the shape it ought to have.
--
Álvaro Herrera 39°49'30"S 73°17'W
Attachments:
v35-libpq-pipeline.patch (text/x-diff, 110.1K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 910e9a81ea..0bffb92462 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3180,6 +3180,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4926,6 +4953,473 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be
+ sent/received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch the connection
+ into pipeline mode,
+ which is done with <xref linkend="libpq-PQenterPipelineMode"/>.
+ <xref linkend="libpq-PQpipelineStatus"/> can be used
+ to test whether pipeline mode is active.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using synchronous command execution functions
+ such as <function>PQfn</function>,
+ <function>PQexec</function>,
+ <function>PQexecParams</function>,
+ <function>PQprepare</function>,
+ <function>PQexecPrepared</function>,
+ <function>PQdescribePrepared</function>,
+ <function>PQdescribePortal</function>,
+ is an error condition.
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQuery"/>,
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave"/>.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ until <literal>PGRES_PIPELINE_SYNC</literal>;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function> will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term><function>PQpipelineStatus</function><indexterm><primary>PQpipelineStatus</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term><function>PQenterPipelineMode</function><indexterm><primary>PQenterPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term><function>PQexitPipelineMode</function><indexterm><primary>PQexitPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term><function>PQsendPipeline</function><indexterm><primary>PQsendPipeline</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4966,6 +5460,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 299d93b241..5dd1e9e936 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1110,6 +1110,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1270,6 +1276,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6a214669c..b7400708fc 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -395,10 +395,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -530,7 +531,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2568,6 +2571,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2757,11 +2764,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2805,8 +2826,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2874,6 +2897,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3127,13 +3156,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3273,7 +3325,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3516,6 +3576,49 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4725,7 +4828,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 4e21057d0f..a714e9fc53 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -522,6 +522,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -553,6 +570,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2459,6 +2482,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3917,6 +3941,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4084,8 +4109,6 @@ freePGconn(PGconn *conn)
if (conn->connip)
free(conn->connip);
/* Note that conn->Pfdebug is not ours to close or free */
- if (conn->last_query)
- free(conn->last_query);
if (conn->write_err_msg)
free(conn->write_err_msg);
if (conn->inBuffer)
@@ -4174,6 +4197,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6726,6 +6750,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..0e964979bc 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,12 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+void pqCommandQueueAdvance(PGconn *conn);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1179,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1209,9 +1217,15 @@ PQsendQueryContinue(PGconn *conn, const char *query)
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ PGcommandQueueEntry *entry = NULL;
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* check the argument */
if (!query)
{
@@ -1220,37 +1234,75 @@ PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
return 0;
}
- /* construct the outgoing Query message */
- if (pqPutMsgStart('Q', conn) < 0 ||
- pqPuts(query, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
+ /* Send the query message(s) */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- /* error message should be set up already */
- return 0;
+ /* construct the outgoing Query message */
+ if (pqPutMsgStart('Q', conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ {
+ /* error message should be set up already */
+ return 0;
+ }
+
+ /* remember we are using simple query protocol */
+ entry->queryclass = PGQUERY_SIMPLE;
+ /* and remember the query text too, if possible */
+ entry->query = strdup(query);
}
+ else
+ {
+ /*
+ * In pipeline mode, we cannot use the simple protocol, so we send
+ * Parse, Bind, Describe Portal, Execute.
+ */
+ if (pqPutMsgStart('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('B', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('D', conn) < 0 ||
+ pqPutc('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('E', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 4, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
- /* remember we are using simple query protocol */
- conn->queryclass = PGQUERY_SIMPLE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ entry->queryclass = PGQUERY_EXTENDED;
+ entry->query = strdup(query);
+ }
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
- {
- /* error message should be set up already */
- return 0;
- }
+ if (pqPipelineFlush(conn) < 0)
+ goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendPipelineCmd(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
}
/*
@@ -1307,6 +1359,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *entry = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1384,10 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1414,42 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ entry->queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* if insufficient memory, query just winds up NULL */
+ entry->query = strdup(query);
+
+ pqAppendPipelineCmd(conn, entry);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1497,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1519,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1593,16 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *entry;
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1712,38 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ entry->queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
if (command)
- conn->last_query = strdup(command);
- else
- conn->last_query = NULL;
+ entry->query = strdup(command);
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendPipelineCmd(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -1647,8 +1762,9 @@ PQsetSingleRowMode(PGconn *conn)
return 0;
if (conn->asyncStatus != PGASYNC_BUSY)
return 0;
- if (conn->queryclass != PGQUERY_SIMPLE &&
- conn->queryclass != PGQUERY_EXTENDED)
+ if (!conn->cmd_queue_head ||
+ (conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE &&
+ conn->cmd_queue_head->queryclass != PGQUERY_EXTENDED))
return 0;
if (conn->result)
return 0;
@@ -1726,14 +1842,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1922,60 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ /*
+ * For any query type other than simple query protocol, we advance
+ * the command queue here. For simple query protocol, we can get
+ * the READY state multiple times before the command is actually
+ * complete, since the string can contain many queries -- so we
+ * wait till we receive ReadyForQuery.
+ */
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
+ pqCommandQueueAdvance(conn);
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-sync result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server. In all other cases, leave the
+ * queue state change for next time, so that a terminating NULL
+ * result is sent.
+ *
+ * In other words: we don't return a NULL after a pipeline
+ * sync.
+ */
+ if (res && res->resultStatus == PGRES_PIPELINE_SYNC)
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2156,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2326,8 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *entry = NULL;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2335,10 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2347,32 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
-
- /* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
-
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- free(conn->last_query);
- conn->last_query = NULL;
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
}
+ /* remember we are doing a Describe */
+ entry->queryclass = PGQUERY_DESCRIBE;
+
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendPipelineCmd(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -2327,7 +2511,8 @@ PQputCopyEnd(PGconn *conn, const char *errormsg)
* If we sent the COPY command in extended-query mode, we must issue a
* Sync as well.
*/
- if (conn->queryclass != PGQUERY_SIMPLE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
{
if (pqPutMsgStart('S', conn) < 0 ||
pqPutMsgEnd(conn) < 0)
@@ -2541,6 +2726,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2747,365 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqCommandQueueAdvance
+ * Remove one query from the command queue, when we receive
+ * all results from the server that pertain to it.
+ */
+void
+pqCommandQueueAdvance(PGconn *conn)
+{
+ PGcommandQueueEntry *prevquery;
+
+ if (conn->cmd_queue_head == NULL)
+ return;
+
+ /* delink from queue */
+ prevquery = conn->cmd_queue_head;
+ conn->cmd_queue_head = conn->cmd_queue_head->next;
+
+ /* and make it recyclable */
+ prevquery->next = NULL;
+ pqRecyclePipelineCmd(conn, prevquery);
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * XXX rewrite this comment. In pipeline mode but nothing left on the
+ * queue; caller can submit more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /* In non-pipeline mode, we're done here */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding commands from the queue until we
+ * get to the next sync from the server.
+ *
+ * The PGRES_PIPELINE_ABORTED results tell the client that its queries
+ * got aborted.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ Assert(entry->next == NULL);
+
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ /* recyclable entries should not have a follow-on command */
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ {
+ free(entry->query);
+ entry->query = NULL;
+ }
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -2569,7 +3120,7 @@ PQresultStatus(const PGresult *res)
char *
PQresStatus(ExecStatusType status)
{
- if ((unsigned int) status >= sizeof pgresStatus / sizeof pgresStatus[0])
+ if ((unsigned int) status >= lengthof(pgresStatus))
return libpq_gettext("invalid ExecStatusType code");
return pgresStatus[status];
}
@@ -3152,6 +3703,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 2ca8c057b9..38dc1e4efa 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're notionally not-IDLE when in pipeline mode we have
+ * completed processing the results of one query and are waiting
+ * for the next one in the pipeline. In this case, as above, just
+ * wait.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,37 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ {
+ /*
+ * In simple query protocol, advance the command queue
+ * (see PQgetResult).
+ */
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_SIMPLE)
+ pqCommandQueueAdvance(conn);
+ conn->asyncStatus = PGASYNC_IDLE;
+ }
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -238,7 +275,8 @@ pqParseInput3(PGconn *conn)
break;
case '1': /* Parse Complete */
/* If we're doing PQprepare, we're done; else ignore */
- if (conn->queryclass == PGQUERY_PREPARE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_PREPARE)
{
if (conn->result == NULL)
{
@@ -285,7 +323,8 @@ pqParseInput3(PGconn *conn)
conn->inCursor += msgLength;
}
else if (conn->result == NULL ||
- conn->queryclass == PGQUERY_DESCRIBE)
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
/* First 'T' in a query sequence */
if (getRowDescriptions(conn, msgLength))
@@ -318,7 +357,8 @@ pqParseInput3(PGconn *conn)
* instead of PGRES_TUPLES_OK. Otherwise we can just
* ignore this message.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE)
{
if (conn->result == NULL)
{
@@ -450,7 +490,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -477,7 +517,9 @@ getRowDescriptions(PGconn *conn, int msgLength)
* PGresult created by getParamDescriptions, and we should fill data into
* that. Otherwise, create a new, empty PGresult.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (!conn->cmd_queue_head ||
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
if (conn->result)
result = conn->result;
@@ -584,7 +626,9 @@ getRowDescriptions(PGconn *conn, int msgLength)
* If we're doing a Describe, we're done, and ready to pass the result
* back to the client.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if ((!conn->cmd_queue_head) ||
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
conn->asyncStatus = PGASYNC_READY;
return 0;
@@ -875,6 +919,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -931,8 +979,8 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res && conn->cmd_queue_head && conn->cmd_queue_head->query)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
/*
* Now build the "overall" error message for PQresultErrorMessage.
@@ -1851,7 +1899,8 @@ pqEndcopy3(PGconn *conn)
* If we sent the COPY command in extended-query mode, we must issue a
* Sync as well.
*/
- if (conn->queryclass != PGQUERY_SIMPLE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
{
if (pqPutMsgStart('S', conn) < 0 ||
pqPutMsgEnd(conn) < 0)
@@ -1931,6 +1980,9 @@ pqFunctionCall3(PGconn *conn, Oid fnid,
int avail;
int i;
+ /* already validated by PQfn */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
/* PQfn already validated connection state */
if (pqPutMsgStart('F', conn) < 0 || /* function call msg */
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 2f052f61f8..9b02e54f0d 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -305,6 +310,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type */
+ char *query; /* SQL command, or NULL if none/unknown/OOM */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -389,12 +410,11 @@ struct pg_conn
ConnStatusType status;
PGAsyncStatusType asyncStatus;
PGTransactionStatusType xactStatus; /* never changes to ACTIVE */
- PGQueryClass queryclass;
- char *last_query; /* last SQL command, or NULL if unknown */
char last_sqlstate[6]; /* last reported SQLSTATE */
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -407,6 +427,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -795,6 +825,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..8819df4d92
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1160 @@
+/*
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+#ifdef HAVE_SYS_SELECT_H
+#include <sys/select.h>
+#endif
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+
+ fflush(stdout);
+
+ fprintf(stderr, "\n%s:%d: ", progname, line);
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ Assert(fmt[strlen(fmt) - 1] != '\n');
+ fprintf(stderr, "\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush
+ * to work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+#if 0
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+#endif
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+ bool goterror;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+#if 0
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+#endif
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQuery(conn, "SELECT 1.0/g FROM generate_series(3, -1, -1) g") != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ goterror = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_SINGLE_TUPLE:
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ break;
+ case PGRES_FATAL_ERROR:
+ if (strcmp(PQresultErrorField(res, PG_DIAG_SQLSTATE), "22012") != 0)
+ pg_fatal("expected division-by-zero, got: %s",
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ printf("got expected division-by-zero\n");
+ goterror = true;
+ break;
+ default:
+ pg_fatal("got unexpected result %s", PQresStatus(PQresultStatus(res)));
+ }
+ PQclear(res);
+ }
+ if (!goterror)
+ pg_fatal("did not get division-by-zero error");
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ bool expect_null;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ expect_null = false;
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ printf("%d: got NULL result\n", i);
+ if (!expect_null)
+ pg_fatal("did not expect NULL here");
+ expect_null = false;
+ continue;
+ }
+ restype = PQresultStatus(res);
+ printf("%d: got status %s", i, PQresStatus(restype));
+ if (expect_null)
+ pg_fatal("expected NULL");
+ if (restype == PGRES_FATAL_ERROR)
+ printf("; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ printf(": command didn't run because pipeline aborted\n");
+ }
+ else
+ printf("\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ else
+ expect_null = true;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..a12f2dd47b
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,31 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow
+ transaction);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libp_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 49614106dc..44b1a43f30 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,10 +33,11 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
-my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
-my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
-my $contrib_extralibs = undef;
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
+my @contrib_uselibpgport = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my @contrib_uselibpgcommon = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my $contrib_extralibs = { 'libpq_pipeline' => ['ws2_32.lib'] };
my $contrib_extraincludes = { 'dblink' => ['src/backend'] };
my $contrib_extrasource = {
'cube' => [ 'contrib/cube/cubescan.l', 'contrib/cube/cubeparse.y' ],
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 574a8a94fa..ca68e88a72 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1562,10 +1562,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 20:51 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 21:40 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-06 00:35 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-11 16:00 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-11 18:49 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-03-13 04:28 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Tom Lane @ 2021-03-11 18:49 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> [ v35-libpq-pipeline.patch ]
I think the changes in pqParseInput3() are broken. You should have
kept the else-structure as-is and inserted the check for "not really
idle" inside the else-clause that reports an error. As it stands,
after successfully processing an asynchronously-received error or
ParameterStatus message, the added code will cause us to return without
advancing inStart, creating an infinite loop of reprocessing that message.
It's possible that we should redefine the way things happen so that if
we're waiting for another pipeline event, we should hold off processing
of async error & ParameterStatus; but in that case you should have added
the pre-emptive return ahead of that if/else structure, where the existing
"If not IDLE state, just wait ..." test is. My guess though is that we
do need to process error messages in that state, so that the correct
patch looks more like
else
{
+ /*
+ * We're notionally not-IDLE when in pipeline mode we have
+ * completed processing the results of one query and are waiting
+ * for the next one in the pipeline. In this case, as above, just
+ * wait.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
pqInternalNotice(&conn->noticeHooks,
"message type 0x%02x arrived from server while idle",
id);
/* Discard the unexpected message */
conn->inCursor += msgLength;
}
It'd be appropriate to do more than nothing to the comment block above
this if/else chain, too, because really that one ought to explain why we
should consume ERROR when in pipeline state.
(I've not looked at the rest of this patch, just scanned what you did
in fe-protocol3.c, because I wondered if there would be any interaction
with the where-to-advance-inStart changes I'm about to commit. Looks
okay modulo this issue.)
regards, tom lane
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 20:51 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 21:40 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-06 00:35 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-11 16:00 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-11 18:49 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Tom Lane <tgl@sss.pgh.pa.us>
@ 2021-03-13 04:28 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-15 18:54 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-13 04:28 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>
On 2021-Mar-11, Tom Lane wrote:
> I think the changes in pqParseInput3() are broken. You should have
> kept the else-structure as-is and inserted the check for "not really
> idle" inside the else-clause that reports an error. As it stands,
> after successfully processing an asynchronously-received error or
> ParameterStatus message, the added code will cause us to return without
> advancing inStart, creating an infinite loop of reprocessing that message.
>
> It's possible that we should redefine the way things happen so that if
> we're waiting for another pipeline event, we should hold off processing
> of async error & ParameterStatus; but in that case you should have added
> the pre-emptive return ahead of that if/else structure, where the existing
> "If not IDLE state, just wait ..." test is.
I think I agree that holding off 'E' and 'S' messages when in between
processing results for different queries in a pipeline, so keeping the
original if/else structure is correct. An error would be correctly
dealt with in the BUSY state immediately afterwards; and the fact that
we pass 'inError' false at that point causes the wrong reaction (namely
that the pipeline is not put in aborted state).
I made a number of other changes: documentation adjustments per comments
from David Johnston, some function renaming as previously noted, and
added test code for PQsendDescribePrepared, PQsendDescribePortal. Also
rebased on latest changes. I also absorbed one change that I already
had when I submitted v35, but hadn't done "git add" on (which caused a
compile failure for CF bot).
--
Álvaro Herrera Valdivia, Chile
Attachments:
v36-libpq-pipeline.patch (text/x-diff, 114.4K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 910e9a81ea..7b938c106c 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3180,6 +3180,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4926,6 +4953,479 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be
+ sent/received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <para>
+ While the pipeline API was introduced in
+ <productname>PostgreSQL</productname> 14, it is a client-side feature
+ which doesn't require special server support, and works on any server
+ that supports the v3 extended query protocol.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch the connection
+ into pipeline mode,
+ which is done with <xref linkend="libpq-PQenterPipelineMode"/>.
+ <xref linkend="libpq-PQpipelineStatus"/> can be used
+ to test whether pipeline mode is active.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using synchronous command execution functions
+ such as <function>PQfn</function>,
+ <function>PQexec</function>,
+ <function>PQexecParams</function>,
+ <function>PQprepare</function>,
+ <function>PQexecPrepared</function>,
+ <function>PQdescribePrepared</function>,
+ <function>PQdescribePortal</function>,
+ is an error condition.
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQuery"/>,
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and does not execute any subsequent command in the queue
+ until the next synchronization point established by
+ <function>PQsendPipeline</function>;
+ a <literal>PGRES_PIPELINE_ABORTED</literal> result is produced for
+ each such command.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave"/>.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ stream result for the first error and all subsequent results
+ until <literal>PGRES_PIPELINE_SYNC</literal>;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ From the client's perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function> will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behavior holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions. If a pipeline synchronization point
+ occurs with an explicit transaction block in aborted state, the next pipeline
+ will become aborted immediately unless the next command puts the transaction
+ in normal mode with <command>ROLLBACK</command>.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its corresponding results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term><function>PQpipelineStatus</function><indexterm><primary>PQpipelineStatus</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term><function>PQenterPipelineMode</function><indexterm><primary>PQenterPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term><function>PQexitPipelineMode</function><indexterm><primary>PQexitPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing,
+ or <function>PQgetResult</function> has not been called to collect
+ results from all previously sent query, returns 0 (in which case,
+ use <xref linkend="libpq-PQerrorMessage"/> to get more information
+ about the failure).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term><function>PQsendPipeline</function><indexterm><primary>PQsendPipeline</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4966,6 +5466,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 299d93b241..5dd1e9e936 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1110,6 +1110,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1270,6 +1276,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index 008a75d207..c9d9900693 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -929,6 +929,8 @@ should_processing_continue(PGresult *res)
case PGRES_COPY_IN:
case PGRES_COPY_BOTH:
case PGRES_SINGLE_TUPLE:
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
return false;
}
return true;
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6a214669c..b7400708fc 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -395,10 +395,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -530,7 +531,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2568,6 +2571,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2757,11 +2764,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2805,8 +2826,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2874,6 +2897,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3127,13 +3156,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3273,7 +3325,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3516,6 +3576,49 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4725,7 +4828,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 4e21057d0f..53b354abb2 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -522,6 +522,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcmdQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcmdQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcmdQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -553,6 +570,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2459,6 +2482,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3917,6 +3941,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4084,8 +4109,6 @@ freePGconn(PGconn *conn)
if (conn->connip)
free(conn->connip);
/* Note that conn->Pfdebug is not ours to close or free */
- if (conn->last_query)
- free(conn->last_query);
if (conn->write_err_msg)
free(conn->write_err_msg);
if (conn->inBuffer)
@@ -4174,6 +4197,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6726,6 +6750,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..ab867b5f33 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,8 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1175,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1184,6 +1188,87 @@ fail:
}
+/*
+ * pqAllocCmdQueueEntry
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendCmdQueueEntry) once the struct is filled in, or
+ * releasing the memory (pqRecycleCmdQueueEntry) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcmdQueueEntry *
+pqAllocCmdQueueEntry(PGconn *conn)
+{
+ PGcmdQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcmdQueueEntry *) malloc(sizeof(PGcmdQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendCmdQueueEntry
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendCmdQueueEntry(PGconn *conn, PGcmdQueueEntry *entry)
+{
+ Assert(entry->next == NULL);
+
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecycleCmdQueueEntry
+ * Push a command queue entry onto the freelist.
+ */
+static void
+pqRecycleCmdQueueEntry(PGconn *conn, PGcmdQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ /* recyclable entries should not have a follow-on command */
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ {
+ free(entry->query);
+ entry->query = NULL;
+ }
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+
/*
* PQsendQuery
* Submit a query, but don't wait for it to finish
@@ -1209,9 +1294,15 @@ PQsendQueryContinue(PGconn *conn, const char *query)
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ PGcmdQueueEntry *entry = NULL;
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* check the argument */
if (!query)
{
@@ -1220,37 +1311,75 @@ PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
return 0;
}
- /* construct the outgoing Query message */
- if (pqPutMsgStart('Q', conn) < 0 ||
- pqPuts(query, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
+ /* Send the query message(s) */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- /* error message should be set up already */
- return 0;
+ /* construct the outgoing Query message */
+ if (pqPutMsgStart('Q', conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ {
+ /* error message should be set up already */
+ return 0;
+ }
+
+ /* remember we are using simple query protocol */
+ entry->queryclass = PGQUERY_SIMPLE;
+ /* and remember the query text too, if possible */
+ entry->query = strdup(query);
}
+ else
+ {
+ /*
+ * In pipeline mode we cannot use the simple protocol, so we send
+ * Parse, Bind, Describe Portal, Execute.
+ */
+ if (pqPutMsgStart('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('B', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('D', conn) < 0 ||
+ pqPutc('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('E', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 4, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
- /* remember we are using simple query protocol */
- conn->queryclass = PGQUERY_SIMPLE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ entry->queryclass = PGQUERY_EXTENDED;
+ entry->query = strdup(query);
+ }
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
- {
- /* error message should be set up already */
- return 0;
- }
+ if (pqPipelineFlush(conn) < 0)
+ goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendCmdQueueEntry(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
+
+sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
+ /* error message should be set up already */
+ return 0;
}
/*
@@ -1307,6 +1436,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcmdQueueEntry *entry = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1461,10 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1491,38 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* Add a Sync, unless in pipeline mode. */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ entry->queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* if insufficient memory, query just winds up NULL */
+ entry->query = strdup(query);
+
+ pqAppendCmdQueueEntry(conn, entry);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1570,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1592,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1666,16 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcmdQueueEntry *entry;
+
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1785,38 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ entry->queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
if (command)
- conn->last_query = strdup(command);
- else
- conn->last_query = NULL;
+ entry->query = strdup(command);
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendCmdQueueEntry(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -1647,8 +1835,9 @@ PQsetSingleRowMode(PGconn *conn)
return 0;
if (conn->asyncStatus != PGASYNC_BUSY)
return 0;
- if (conn->queryclass != PGQUERY_SIMPLE &&
- conn->queryclass != PGQUERY_EXTENDED)
+ if (!conn->cmd_queue_head ||
+ (conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE &&
+ conn->cmd_queue_head->queryclass != PGQUERY_EXTENDED))
return 0;
if (conn->result)
return 0;
@@ -1726,14 +1915,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1995,61 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+
+ /*
+ * For any query type other than simple query protocol, we advance
+ * the command queue here. For simple query protocol, we can get
+ * the READY state multiple times before the command is actually
+ * complete, since the command string can contain many queries --
+ * so we wait till we receive ReadyForQuery.
+ */
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
+ pqCommandQueueAdvance(conn);
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-sync result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server. In all other cases, leave the
+ * queue state change for next time, so that a terminating
+ * NULL result is sent.
+ *
+ * In other words: we don't return a NULL after a pipeline
+ * sync.
+ */
+ if (res && res->resultStatus == PGRES_PIPELINE_SYNC)
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2230,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2400,8 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcmdQueueEntry *entry = NULL;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2409,10 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2421,32 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
-
- /* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
-
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- free(conn->last_query);
- conn->last_query = NULL;
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
}
+ /* remember we are doing a Describe */
+ entry->queryclass = PGQUERY_DESCRIBE;
+
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendCmdQueueEntry(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -2327,7 +2585,8 @@ PQputCopyEnd(PGconn *conn, const char *errormsg)
* If we sent the COPY command in extended-query mode, we must issue a
* Sync as well.
*/
- if (conn->queryclass != PGQUERY_SIMPLE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
{
if (pqPutMsgStart('S', conn) < 0 ||
pqPutMsgEnd(conn) < 0)
@@ -2541,6 +2800,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2821,285 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqCommandQueueAdvance
+ * Remove one query from the command queue, when we receive
+ * all results from the server that pertain to it.
+ */
+void
+pqCommandQueueAdvance(PGconn *conn)
+{
+ PGcmdQueueEntry *prevquery;
+
+ if (conn->cmd_queue_head == NULL)
+ return;
+
+ /* delink from queue */
+ prevquery = conn->cmd_queue_head;
+ conn->cmd_queue_head = conn->cmd_queue_head->next;
+
+ /* and make it recyclable */
+ prevquery->next = NULL;
+ pqRecycleCmdQueueEntry(conn, prevquery);
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * XXX rewrite this comment. In pipeline mode but nothing left on the
+ * queue; caller can submit more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /* In non-pipeline mode, we're done here */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding commands from the queue until we
+ * get to the next sync from the server.
+ *
+ * The PGRES_PIPELINE_ABORTED results tell the client that its queries
+ * got aborted.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcmdQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendCmdQueueEntry(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -2569,7 +3114,7 @@ PQresultStatus(const PGresult *res)
char *
PQresStatus(ExecStatusType status)
{
- if ((unsigned int) status >= sizeof pgresStatus / sizeof pgresStatus[0])
+ if ((unsigned int) status >= lengthof(pgresStatus))
return libpq_gettext("invalid ExecStatusType code");
return pgresStatus[status];
}
@@ -3152,6 +3697,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index eb55d528fb..306e89acfd 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -158,6 +158,18 @@ pqParseInput3(PGconn *conn)
if (conn->asyncStatus != PGASYNC_IDLE)
return;
+ /*
+ * We're also notionally not-IDLE when in pipeline mode the state
+ * says "idle" (so we have completed receiving the results of one
+ * query from the server and dispatched them to the application)
+ * but another query is queued; yield back control to caller so
+ * that they can initiate processing of the next query in the
+ * queue.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
/*
* Unexpected message in IDLE state; need to recover somehow.
* ERROR messages are handled using the notice processor;
@@ -179,6 +191,7 @@ pqParseInput3(PGconn *conn)
}
else
{
+ /* Any other case is unexpected and we summarily skip it */
pqInternalNotice(&conn->noticeHooks,
"message type 0x%02x arrived from server while idle",
id);
@@ -217,10 +230,37 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ {
+ /*
+ * In simple query protocol, advance the command queue
+ * (see PQgetResult).
+ */
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_SIMPLE)
+ pqCommandQueueAdvance(conn);
+ conn->asyncStatus = PGASYNC_IDLE;
+ }
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -238,7 +278,8 @@ pqParseInput3(PGconn *conn)
break;
case '1': /* Parse Complete */
/* If we're doing PQprepare, we're done; else ignore */
- if (conn->queryclass == PGQUERY_PREPARE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_PREPARE)
{
if (conn->result == NULL)
{
@@ -285,7 +326,8 @@ pqParseInput3(PGconn *conn)
conn->inCursor += msgLength;
}
else if (conn->result == NULL ||
- conn->queryclass == PGQUERY_DESCRIBE)
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
/* First 'T' in a query sequence */
if (getRowDescriptions(conn, msgLength))
@@ -316,7 +358,8 @@ pqParseInput3(PGconn *conn)
* instead of PGRES_TUPLES_OK. Otherwise we can just
* ignore this message.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE)
{
if (conn->result == NULL)
{
@@ -445,7 +488,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -471,7 +514,9 @@ getRowDescriptions(PGconn *conn, int msgLength)
* PGresult created by getParamDescriptions, and we should fill data into
* that. Otherwise, create a new, empty PGresult.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (!conn->cmd_queue_head ||
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
if (conn->result)
result = conn->result;
@@ -568,7 +613,9 @@ getRowDescriptions(PGconn *conn, int msgLength)
* If we're doing a Describe, we're done, and ready to pass the result
* back to the client.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if ((!conn->cmd_queue_head) ||
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
conn->asyncStatus = PGASYNC_READY;
return 0;
@@ -841,6 +888,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -897,8 +948,8 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res && conn->cmd_queue_head && conn->cmd_queue_head->query)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
/*
* Now build the "overall" error message for PQresultErrorMessage.
@@ -1817,7 +1868,8 @@ pqEndcopy3(PGconn *conn)
* If we sent the COPY command in extended-query mode, we must issue a
* Sync as well.
*/
- if (conn->queryclass != PGQUERY_SIMPLE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
{
if (pqPutMsgStart('S', conn) < 0 ||
pqPutMsgEnd(conn) < 0)
@@ -1897,6 +1949,9 @@ pqFunctionCall3(PGconn *conn, Oid fnid,
int avail;
int i;
+ /* already validated by PQfn */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
/* PQfn already validated connection state */
if (pqPutMsgStart('F', conn) < 0 || /* function call msg */
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..fb31a49fab 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 2f052f61f8..60f604d685 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,21 +217,16 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
} PGAsyncStatusType;
-/* PGQueryClass tracks which query protocol we are now executing */
-typedef enum
-{
- PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
- PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
- PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
-} PGQueryClass;
-
/* Target server type (decoded value of target_session_attrs) */
typedef enum
{
@@ -305,6 +300,29 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/*
+ * PGQueryClass tracks which query protocol is in use for each command queue
+ * entry, or special operation in execution
+ */
+typedef enum
+{
+ PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
+ PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
+ PGQUERY_PREPARE, /* Parse only (PQprepare) */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync (at end of a pipeline) */
+} PGQueryClass;
+
+/*
+ * An entry in the pending command queue.
+ */
+typedef struct PGcmdQueueEntry
+{
+ PGQueryClass queryclass; /* Query type */
+ char *query; /* SQL command, or NULL if none/unknown/OOM */
+ struct PGcmdQueueEntry *next; /* list link */
+} PGcmdQueueEntry;
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -389,12 +407,11 @@ struct pg_conn
ConnStatusType status;
PGAsyncStatusType asyncStatus;
PGTransactionStatusType xactStatus; /* never changes to ACTIVE */
- PGQueryClass queryclass;
- char *last_query; /* last SQL command, or NULL if unknown */
char last_sqlstate[6]; /* last reported SQLSTATE */
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -407,6 +424,19 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The pending command queue as a singly-linked list. Head is the command
+ * currently in execution, tail is where new commands are added.
+ */
+ PGcmdQueueEntry *cmd_queue_head;
+ PGcmdQueueEntry *cmd_queue_tail;
+
+ /*
+ * To save malloc traffic, we don't free entries right away; instead we
+ * save them in this list for possible reuse.
+ */
+ PGcmdQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -622,6 +652,7 @@ extern void pqSaveMessageField(PGresult *res, char code,
extern void pqSaveParameterStatus(PGconn *conn, const char *name,
const char *value);
extern int pqRowProcessor(PGconn *conn, const char **errmsgp);
+extern void pqCommandQueueAdvance(PGconn *conn);
extern int PQsendQueryContinue(PGconn *conn, const char *query);
/* === in fe-protocol3.c === */
@@ -795,6 +826,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..cb79979a95
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1263 @@
+/*-------------------------------------------------------------------------
+ *
+ * libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+#ifdef HAVE_SYS_SELECT_H
+#include <sys/select.h>
+#endif
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+
+ fflush(stdout);
+
+ fprintf(stderr, "\n%s:%d: ", progname, line);
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ Assert(fmt[strlen(fmt) - 1] != '\n');
+ fprintf(stderr, "\n");
+ exit(1);
+}
+
+static void
+test_disallowed_in_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_pipeline_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+ bool goterror;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQuery(conn, "SELECT 1.0/g FROM generate_series(3, -1, -1) g") != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ PQsetSingleRowMode(conn);
+ goterror = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_SINGLE_TUPLE:
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ break;
+ case PGRES_FATAL_ERROR:
+ if (strcmp(PQresultErrorField(res, PG_DIAG_SQLSTATE), "22012") != 0)
+ pg_fatal("expected division-by-zero, got: %s",
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ printf("got expected division-by-zero\n");
+ goterror = true;
+ break;
+ default:
+ pg_fatal("got unexpected result %s", PQresStatus(PQresultStatus(res)));
+ }
+ PQclear(res);
+ }
+ if (!goterror)
+ pg_fatal("did not get division-by-zero error");
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+};
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ enum PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_prepared(PGconn *conn)
+{
+ PGresult *res = NULL;
+ Oid param_oids[1] = {INT4OID};
+ Oid expected_oids[4];
+ Oid typ;
+
+ fprintf(stderr, "prepared... ");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "select_one", "SELECT $1, '42', $1::numeric, "
+ "interval '1 sec'",
+ 1, param_oids) != 1)
+ pg_fatal("preparing query failed: %s", PQerrorMessage(conn));
+ expected_oids[0] = INT4OID;
+ expected_oids[1] = TEXTOID;
+ expected_oids[2] = NUMERICOID;
+ expected_oids[3] = INTERVALOID;
+ if (PQsendDescribePrepared(conn, "select_one") != 1)
+ pg_fatal("failed to send describePrepared: %s", PQerrorMessage(conn));
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("failed to send pipeline: %s", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("expected COMMAND_OK, got %s", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("expected NULL result");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned NULL");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("expected COMMAND_OK, got %s", PQresStatus(PQresultStatus(res)));
+ if (PQnfields(res) != lengthof(expected_oids))
+ pg_fatal("expected %d columns, got %d",
+ lengthof(expected_oids), PQnfields(res));
+ for (int i = 0; i < PQnfields(res); i++)
+ {
+ typ = PQftype(res, i);
+ if (typ != expected_oids[i])
+ pg_fatal("field %d: expected type %u, got %u",
+ i, expected_oids[i], typ);
+ }
+ PQclear(res);
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("expected NULL result");
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("expected PGRES_PIPELINE_SYNC, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("could not exit pipeline mode: %s", PQerrorMessage(conn));
+
+ PQexec(conn, "BEGIN");
+ PQexec(conn, "DECLARE cursor_one CURSOR FOR SELECT 1");
+ PQenterPipelineMode(conn);
+ if (PQsendDescribePortal(conn, "cursor_one") != 1)
+ pg_fatal("PQsendDescribePortal failed: %s", PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("expected NULL result");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("expected COMMAND_OK, got %s", PQresStatus(PQresultStatus(res)));
+
+ typ = PQftype(res, 0);
+ if (typ != INT4OID)
+ pg_fatal("portal: expected type %u, got %u",
+ INT4OID, typ);
+ PQclear(res);
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("expected NULL result");
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("expected PGRES_PIPELINE_SYNC, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("could not exit pipeline mode: %s", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush
+ * to work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ bool expect_null;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQsendPipeline above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+ PQsendPipeline(conn);
+ num_sends++;
+
+ expect_null = false;
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ printf("%d: got NULL result\n", i);
+ if (!expect_null)
+ pg_fatal("did not expect NULL here");
+ expect_null = false;
+ continue;
+ }
+ restype = PQresultStatus(res);
+ printf("%d: got status %s", i, PQresStatus(restype));
+ if (expect_null)
+ pg_fatal("expected NULL");
+ if (restype == PGRES_FATAL_ERROR)
+ printf("; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ printf(": command didn't run because pipeline aborted\n");
+ }
+ else
+ printf("\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ else
+ expect_null = true;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s tests", progname);
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+}
+
+static void
+print_test_list(void)
+{
+ printf("disallowed_in_pipeline\n");
+ printf("multi_pipelines\n");
+ printf("pipeline_abort\n");
+ printf("pipelined_insert\n");
+ printf("prepared\n");
+ printf("simple_pipeline\n");
+ printf("singlerow\n");
+ printf("transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ if (strcmp(argv[1], "tests") == 0)
+ {
+ print_test_list();
+ exit(0);
+ }
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed_in_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipelines") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_pipeline_abort(conn);
+ else if (strcmp(argv[1], "pipelined_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "prepared") == 0)
+ test_prepared(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..ba15b64ca7
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,28 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 8;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+my ($out, $err) = run_command(['libpq_pipeline', 'tests']);
+die "oops: $err" unless $err eq '';
+my @tests = split(/\s/, $out);
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libpq_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 74fde40e3a..a184404e21 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,10 +33,11 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
-my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
-my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
-my $contrib_extralibs = undef;
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
+my @contrib_uselibpgport = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my @contrib_uselibpgcommon = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my $contrib_extralibs = { 'libpq_pipeline' => ['ws2_32.lib'] };
my $contrib_extraincludes = { 'dblink' => ['src/backend'] };
my $contrib_extrasource = {
'cube' => [ 'contrib/cube/cubescan.l', 'contrib/cube/cubeparse.y' ],
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e017557e3e..41a62be994 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1563,10 +1563,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcmdQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 20:51 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 21:40 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-06 00:35 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-11 16:00 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-11 18:49 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Tom Lane <tgl@sss.pgh.pa.us>
2021-03-13 04:28 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-15 18:54 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-15 19:00 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-15 18:54 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Álvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Aya Iwata <iwata.aya@jp.fujitsu.com>; Daniel Vérité <daniel@manitou-mail.org>; David G. Johnston <david.g.johnston@gmail.com>; Kirk Jamison <k.jamison@fujitsu.com>; Michael Paquier <michael.paquier@gmail.com>; Nikhil Sontakke <nikhils@2ndquadrant.com>; Vaishnavi Prabakaran <VaishnaviP@fast.au.fujitsu.com>
Here's what seems a final version of the patch. I renamed one more
function: PQsendPipeline is now PQpipelineSync. I also reworded the
docs in a couple of places, added a few tests to the pgbench patch, and
made it work.
Note the pgbench results in pipeline mode:
./pgbench -r -Mextended -n -f /home/alvherre/Code/pgsql-build/pipeline/src/bin/pgbench/tmp_check/t_001_pgbench_with_server_main_data/001_pgbench_pipeline -c 100 -t10000
pgbench (PostgreSQL) 14.0
transaction type: /home/alvherre/Code/pgsql-build/pipeline/src/bin/pgbench/tmp_check/t_001_pgbench_with_server_main_data/001_pgbench_pipeline
scaling factor: 1
query mode: extended
number of clients: 100
number of threads: 1
number of transactions per client: 10000
number of transactions actually processed: 1000000/1000000
latency average = 2.316 ms
initial connection time = 113.859 ms
tps = 43182.438635 (without initial connection time)
statement latencies in milliseconds:
0.000 \startpipeline
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
0.000 select 1;
1.624 \endpipeline
If I just replace the \startpipeline and \endpipeline lines with BEGIN
and COMMIT respectively, I get this:
tps = 10220.259051 (without initial connection time)
0.830 begin;
0.765 select 1;
0.752 select 1;
0.753 select 1;
0.755 select 1;
0.754 select 1;
0.755 select 1;
0.757 select 1;
0.756 select 1;
0.756 select 1;
0.756 select 1;
0.750 commit;
Yes, you could say that this is a liiiitle bit unfair -- but it seems
quite impressive nonetheless.
--
Álvaro Herrera 39°49'30"S 73°17'W
Attachments: download all (2 files, .tar.gz)
v37-0001-Implement-pipeline-mode-in-libpq.patch (text/x-diff, 113.0K) download
From 5e4fdd5246d559caf0d75ad74001f09a48ec4c0e Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 15 Mar 2021 15:05:22 -0300
Subject: [PATCH v37 1/2] Implement pipeline mode in libpq
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Pipeline mode in libpq lets an application avoid the Sync messages in
the FE/BE protocol that are implicit in the old libpq API after each
query. The application can then insert Sync at its leisure with a new
libpq function PQpipelineSync. This can lead to substantial reductions
in query latency.
Co-authored-by: Craig Ringer <craig.ringer@enterprisedb.com>
Co-authored-by: Matthieu Garrigues <matthieu.garrigues@gmail.com>
Co-authored-by: Álvaro Herrera <alvherre@alvh.no-ip.org>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Aya Iwata <iwata.aya@jp.fujitsu.com>
Reviewed-by: Daniel Vérité <daniel@manitou-mail.org>
Reviewed-by: David G. Johnston <david.g.johnston@gmail.com>
Reviewed-by: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-by: Kirk Jamison <k.jamison@fujitsu.com>
Reviewed-by: Michael Paquier <michael.paquier@gmail.com>
Reviewed-by: Nikhil Sontakke <nikhils@2ndquadrant.com>
Reviewed-by: Vaishnavi Prabakaran <VaishnaviP@fast.au.fujitsu.com>
Reviewed-by: Zhihong Yu <zyu@yugabyte.com>
Discussion: https://postgr.es/m/CAMsr+YFUjJytRyV4J-16bEoiZyH=4nj+sQ7JP9ajwz=B4dMMZw@mail.gmail.com
Discussion: https://postgr.es/m/CAJkzx4T5E-2cQe3dtv2R78dYFvz+in8PY7A8MArvLhs_pg75gg@mail.gmail.com
---
doc/src/sgml/libpq.sgml | 522 ++++++-
doc/src/sgml/lobj.sgml | 4 +
doc/src/sgml/ref/pgbench.sgml | 21 +
.../libpqwalreceiver/libpqwalreceiver.c | 6 +
src/bin/pg_amcheck/pg_amcheck.c | 2 +
src/interfaces/libpq/exports.txt | 4 +
src/interfaces/libpq/fe-connect.c | 37 +-
src/interfaces/libpq/fe-exec.c | 717 ++++++++-
src/interfaces/libpq/fe-protocol3.c | 77 +-
src/interfaces/libpq/libpq-fe.h | 21 +-
src/interfaces/libpq/libpq-int.h | 60 +-
src/test/modules/Makefile | 1 +
src/test/modules/libpq_pipeline/.gitignore | 5 +
src/test/modules/libpq_pipeline/Makefile | 20 +
src/test/modules/libpq_pipeline/README | 1 +
.../modules/libpq_pipeline/libpq_pipeline.c | 1303 +++++++++++++++++
.../libpq_pipeline/t/001_libpq_pipeline.pl | 28 +
src/tools/msvc/Mkvcbuild.pm | 9 +-
src/tools/pgindent/typedefs.list | 2 +
19 files changed, 2727 insertions(+), 113 deletions(-)
create mode 100644 src/test/modules/libpq_pipeline/.gitignore
create mode 100644 src/test/modules/libpq_pipeline/Makefile
create mode 100644 src/test/modules/libpq_pipeline/README
create mode 100644 src/test/modules/libpq_pipeline/libpq_pipeline.c
create mode 100644 src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 910e9a81ea..be674fbaa9 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3180,6 +3180,33 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQpipelineSync"/>.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that has
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and each time it will return this status code
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4677,8 +4704,9 @@ int PQsendDescribePortal(PGconn *conn, const char *portalName);
<xref linkend="libpq-PQsendQueryParams"/>,
<xref linkend="libpq-PQsendPrepare"/>,
<xref linkend="libpq-PQsendQueryPrepared"/>,
- <xref linkend="libpq-PQsendDescribePrepared"/>, or
- <xref linkend="libpq-PQsendDescribePortal"/>
+ <xref linkend="libpq-PQsendDescribePrepared"/>,
+ <xref linkend="libpq-PQsendDescribePortal"/>, or
+ <xref linkend="libpq-PQpipelineSync"/>
call, and returns it.
A null pointer is returned when the command is complete and there
will be no more results.
@@ -4702,6 +4730,19 @@ PGresult *PQgetResult(PGconn *conn);
<xref linkend="libpq-PQconsumeInput"/>.
</para>
+ <para>
+ In pipeline mode, <function>PQgetResult</function> will return normally
+ unless an error occurs; for any subsequent query sent after the one
+ that caused the error until (and excluding) the next synchronization point,
+ a special result of type <literal>PGRES_PIPELINE_ABORTED</literal> will
+ be returned, and a null pointer will be returned after it.
+ When the pipeline synchronization point is reached, a result of type
+ <literal>PGRES_PIPELINE_SYNC</literal> will be returned.
+ The result of the next query after the synchronization point follows
+ immediately (that is, no null pointer is returned after
+ the synchronization point.)
+ </para>
+
<note>
<para>
Even when <xref linkend="libpq-PQresultStatus"/> indicates a fatal
@@ -4926,6 +4967,476 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be
+ sent/received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <para>
+ While the pipeline API was introduced in
+ <productname>PostgreSQL</productname> 14, it is a client-side feature
+ which doesn't require special server support, and works on any server
+ that supports the v3 extended query protocol.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines, the application must switch the connection
+ into pipeline mode,
+ which is done with <xref linkend="libpq-PQenterPipelineMode"/>.
+ <xref linkend="libpq-PQpipelineStatus"/> can be used
+ to test whether pipeline mode is active.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using synchronous command execution functions
+ such as <function>PQfn</function>,
+ <function>PQexec</function>,
+ <function>PQexecParams</function>,
+ <function>PQprepare</function>,
+ <function>PQexecPrepared</function>,
+ <function>PQdescribePrepared</function>,
+ <function>PQdescribePortal</function>,
+ is an error condition.
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before it switches to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQuery"/>,
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQpipelineSync"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and does not execute any subsequent command in the queue
+ until the next synchronization point established by
+ <function>PQpipelineSync</function>;
+ a <literal>PGRES_PIPELINE_ABORTED</literal> result is produced for
+ each such command.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one query in a pipeline, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave"/>.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQpipelineSync</function> at the corresponding point
+ in the pipeline.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ query result for the first error and all subsequent results
+ until the next <literal>PGRES_PIPELINE_SYNC</literal>;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ From the client's perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function> will report a
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQpipelineSync</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
+ the failed operation are skipped entirely. The same behavior holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions. If a pipeline synchronization point
+ occurs with an explicit transaction block in aborted state, the next pipeline
+ will become aborted immediately unless the next command puts the transaction
+ in normal mode with <command>ROLLBACK</command>.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ remaining to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its corresponding results queue. Based on available memory, results from the
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/libpq_pipeline/libpq_pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term><function>PQpipelineStatus</function><indexterm><primary>PQpipelineStatus</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQgetResult</function>
+ returns a result of type <literal>PGRES_PIPELINE_SYNC</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term><function>PQenterPipelineMode</function><indexterm><primary>PQenterPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term><function>PQexitPipelineMode</function><indexterm><primary>PQexitPipelineMode</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing,
+ or <function>PQgetResult</function> has not been called to collect
+ results from all previously sent query, returns 0 (in which case,
+ use <xref linkend="libpq-PQerrorMessage"/> to get more information
+ about the failure).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQpipelineSync">
+ <term><function>PQpipelineSync</function><indexterm><primary>PQpipelineSync</primary></indexterm></term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQpipelineSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid succession. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ would have to introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4966,6 +5477,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 299d93b241..5dd1e9e936 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1110,6 +1110,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1270,6 +1276,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pg_amcheck/pg_amcheck.c b/src/bin/pg_amcheck/pg_amcheck.c
index 008a75d207..c9d9900693 100644
--- a/src/bin/pg_amcheck/pg_amcheck.c
+++ b/src/bin/pg_amcheck/pg_amcheck.c
@@ -929,6 +929,8 @@ should_processing_continue(PGresult *res)
case PGRES_COPY_IN:
case PGRES_COPY_BOTH:
case PGRES_SINGLE_TUPLE:
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
return false;
}
return true;
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..5c48c14191 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQpipelineSync 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 4e21057d0f..53b354abb2 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -522,6 +522,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcmdQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcmdQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcmdQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -553,6 +570,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2459,6 +2482,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -3917,6 +3941,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4084,8 +4109,6 @@ freePGconn(PGconn *conn)
if (conn->connip)
free(conn->connip);
/* Note that conn->Pfdebug is not ours to close or free */
- if (conn->last_query)
- free(conn->last_query);
if (conn->write_err_msg)
free(conn->write_err_msg);
if (conn->inBuffer)
@@ -4174,6 +4197,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6726,6 +6750,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9a038043b2..f3443708a6 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,8 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1175,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1184,6 +1188,87 @@ fail:
}
+/*
+ * pqAllocCmdQueueEntry
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendCmdQueueEntry) once the struct is filled in, or
+ * releasing the memory (pqRecycleCmdQueueEntry) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcmdQueueEntry *
+pqAllocCmdQueueEntry(PGconn *conn)
+{
+ PGcmdQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcmdQueueEntry *) malloc(sizeof(PGcmdQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendCmdQueueEntry
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendCmdQueueEntry(PGconn *conn, PGcmdQueueEntry *entry)
+{
+ Assert(entry->next == NULL);
+
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecycleCmdQueueEntry
+ * Push a command queue entry onto the freelist.
+ */
+static void
+pqRecycleCmdQueueEntry(PGconn *conn, PGcmdQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ /* recyclable entries should not have a follow-on command */
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ {
+ free(entry->query);
+ entry->query = NULL;
+ }
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+
/*
* PQsendQuery
* Submit a query, but don't wait for it to finish
@@ -1209,9 +1294,15 @@ PQsendQueryContinue(PGconn *conn, const char *query)
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ PGcmdQueueEntry *entry = NULL;
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* check the argument */
if (!query)
{
@@ -1220,37 +1311,75 @@ PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
return 0;
}
- /* construct the outgoing Query message */
- if (pqPutMsgStart('Q', conn) < 0 ||
- pqPuts(query, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
+ /* Send the query message(s) */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- /* error message should be set up already */
- return 0;
+ /* construct the outgoing Query message */
+ if (pqPutMsgStart('Q', conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ {
+ /* error message should be set up already */
+ return 0;
+ }
+
+ /* remember we are using simple query protocol */
+ entry->queryclass = PGQUERY_SIMPLE;
+ /* and remember the query text too, if possible */
+ entry->query = strdup(query);
}
+ else
+ {
+ /*
+ * In pipeline mode we cannot use the simple protocol, so we send
+ * Parse, Bind, Describe Portal, Execute.
+ */
+ if (pqPutMsgStart('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts(query, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('B', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutInt(0, 2, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('D', conn) < 0 ||
+ pqPutc('P', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ if (pqPutMsgStart('E', conn) < 0 ||
+ pqPuts("", conn) < 0 ||
+ pqPutInt(0, 4, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
- /* remember we are using simple query protocol */
- conn->queryclass = PGQUERY_SIMPLE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ entry->queryclass = PGQUERY_EXTENDED;
+ entry->query = strdup(query);
+ }
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
- {
- /* error message should be set up already */
- return 0;
- }
+ if (pqPipelineFlush(conn) < 0)
+ goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendCmdQueueEntry(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
+
+sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
+ /* error message should be set up already */
+ return 0;
}
/*
@@ -1307,6 +1436,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcmdQueueEntry *entry = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1330,6 +1461,10 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1356,32 +1491,38 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* Add a Sync, unless in pipeline mode. */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ entry->queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* if insufficient memory, query just winds up NULL */
+ entry->query = strdup(query);
+
+ pqAppendCmdQueueEntry(conn, entry);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -1429,7 +1570,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1450,20 +1592,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1487,10 +1666,16 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcmdQueueEntry *entry;
+
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1600,35 +1785,38 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ entry->queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
if (command)
- conn->last_query = strdup(command);
- else
- conn->last_query = NULL;
+ entry->query = strdup(command);
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendCmdQueueEntry(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -1647,8 +1835,9 @@ PQsetSingleRowMode(PGconn *conn)
return 0;
if (conn->asyncStatus != PGASYNC_BUSY)
return 0;
- if (conn->queryclass != PGQUERY_SIMPLE &&
- conn->queryclass != PGQUERY_EXTENDED)
+ if (!conn->cmd_queue_head ||
+ (conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE &&
+ conn->cmd_queue_head->queryclass != PGQUERY_EXTENDED))
return 0;
if (conn->result)
return 0;
@@ -1726,14 +1915,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1803,8 +1995,62 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round of
+ * results from the current query; prepare to send the results
+ * of the next query when we're called next. Also, since this
+ * is the start of the results of the next query, clear any
+ * prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+
+ /*
+ * For any query type other than simple query protocol, we advance
+ * the command queue here. This is because for simple query
+ * protocol we can get the READY state multiple times before the
+ * command is actually complete, since the command string can
+ * contain many queries. In simple query protocol, the queue
+ * advance is done by fe-protocol3 when it receives ReadyForQuery.
+ */
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
+ pqCommandQueueAdvance(conn);
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle now, and ...
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /*
+ * ... in cases when we're sending a pipeline-sync result,
+ * move queue processing forwards immediately, so that next
+ * time we're called, we're prepared to return the next result
+ * received from the server. In all other cases, leave the
+ * queue state change for next time, so that a terminating
+ * NULL result is sent.
+ *
+ * (In other words: we don't return a NULL after a pipeline
+ * sync.)
+ */
+ if (res && res->resultStatus == PGRES_PIPELINE_SYNC)
+ pqPipelineProcessQueue(conn);
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1985,6 +2231,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2148,6 +2401,8 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcmdQueueEntry *entry = NULL;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2155,6 +2410,10 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
if (!PQsendQueryStart(conn, true))
return 0;
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
/* construct the Describe message */
if (pqPutMsgStart('D', conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2163,32 +2422,32 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
-
- /* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
-
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
{
- free(conn->last_query);
- conn->last_query = NULL;
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
}
+ /* remember we are doing a Describe */
+ entry->queryclass = PGQUERY_DESCRIBE;
+
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ pqAppendCmdQueueEntry(conn, entry);
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
/* error message should be set up already */
return 0;
}
@@ -2327,7 +2586,8 @@ PQputCopyEnd(PGconn *conn, const char *errormsg)
* If we sent the COPY command in extended-query mode, we must issue a
* Sync as well.
*/
- if (conn->queryclass != PGQUERY_SIMPLE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
{
if (pqPutMsgStart('S', conn) < 0 ||
pqPutMsgEnd(conn) < 0)
@@ -2541,6 +2801,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2555,6 +2822,277 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQpipelineSync. Multiple sync
+ * points can be established while in pipeline mode. Pipeline mode can
+ * be exited by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqCommandQueueAdvance
+ * Remove one query from the command queue, when we receive
+ * all results from the server that pertain to it.
+ */
+void
+pqCommandQueueAdvance(PGconn *conn)
+{
+ PGcmdQueueEntry *prevquery;
+
+ if (conn->cmd_queue_head == NULL)
+ return;
+
+ /* delink from queue */
+ prevquery = conn->cmd_queue_head;
+ conn->cmd_queue_head = conn->cmd_queue_head->next;
+
+ /* and make it recyclable */
+ prevquery->next = NULL;
+ pqRecycleCmdQueueEntry(conn, prevquery);
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ /* Nothing to do if not in pipeline mode, or queue is empty */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF ||
+ conn->cmd_queue_head == NULL)
+ return;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding commands from the queue until we
+ * get to the next sync from the server.
+ *
+ * The PGRES_PIPELINE_ABORTED results tell the client that its queries
+ * got aborted.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQpipelineSync
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQpipelineSync gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQpipelineSync is called, but
+ * PQpipelineSync need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQpipelineSync(PGconn *conn)
+{
+ PGcmdQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqAllocCmdQueueEntry(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendCmdQueueEntry(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecycleCmdQueueEntry(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -2569,7 +3107,7 @@ PQresultStatus(const PGresult *res)
char *
PQresStatus(ExecStatusType status)
{
- if ((unsigned int) status >= sizeof pgresStatus / sizeof pgresStatus[0])
+ if ((unsigned int) status >= lengthof(pgresStatus))
return libpq_gettext("invalid ExecStatusType code");
return pgresStatus[status];
}
@@ -3152,6 +3690,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus != PQ_PIPELINE_ON) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return pqFlush(conn);
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index eb55d528fb..306e89acfd 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -158,6 +158,18 @@ pqParseInput3(PGconn *conn)
if (conn->asyncStatus != PGASYNC_IDLE)
return;
+ /*
+ * We're also notionally not-IDLE when in pipeline mode the state
+ * says "idle" (so we have completed receiving the results of one
+ * query from the server and dispatched them to the application)
+ * but another query is queued; yield back control to caller so
+ * that they can initiate processing of the next query in the
+ * queue.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
/*
* Unexpected message in IDLE state; need to recover somehow.
* ERROR messages are handled using the notice processor;
@@ -179,6 +191,7 @@ pqParseInput3(PGconn *conn)
}
else
{
+ /* Any other case is unexpected and we summarily skip it */
pqInternalNotice(&conn->noticeHooks,
"message type 0x%02x arrived from server while idle",
id);
@@ -217,10 +230,37 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ {
+ /*
+ * In simple query protocol, advance the command queue
+ * (see PQgetResult).
+ */
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_SIMPLE)
+ pqCommandQueueAdvance(conn);
+ conn->asyncStatus = PGASYNC_IDLE;
+ }
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -238,7 +278,8 @@ pqParseInput3(PGconn *conn)
break;
case '1': /* Parse Complete */
/* If we're doing PQprepare, we're done; else ignore */
- if (conn->queryclass == PGQUERY_PREPARE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_PREPARE)
{
if (conn->result == NULL)
{
@@ -285,7 +326,8 @@ pqParseInput3(PGconn *conn)
conn->inCursor += msgLength;
}
else if (conn->result == NULL ||
- conn->queryclass == PGQUERY_DESCRIBE)
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
/* First 'T' in a query sequence */
if (getRowDescriptions(conn, msgLength))
@@ -316,7 +358,8 @@ pqParseInput3(PGconn *conn)
* instead of PGRES_TUPLES_OK. Otherwise we can just
* ignore this message.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE)
{
if (conn->result == NULL)
{
@@ -445,7 +488,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -471,7 +514,9 @@ getRowDescriptions(PGconn *conn, int msgLength)
* PGresult created by getParamDescriptions, and we should fill data into
* that. Otherwise, create a new, empty PGresult.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if (!conn->cmd_queue_head ||
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
if (conn->result)
result = conn->result;
@@ -568,7 +613,9 @@ getRowDescriptions(PGconn *conn, int msgLength)
* If we're doing a Describe, we're done, and ready to pass the result
* back to the client.
*/
- if (conn->queryclass == PGQUERY_DESCRIBE)
+ if ((!conn->cmd_queue_head) ||
+ (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass == PGQUERY_DESCRIBE))
{
conn->asyncStatus = PGASYNC_READY;
return 0;
@@ -841,6 +888,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
@@ -897,8 +948,8 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res && conn->cmd_queue_head && conn->cmd_queue_head->query)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
/*
* Now build the "overall" error message for PQresultErrorMessage.
@@ -1817,7 +1868,8 @@ pqEndcopy3(PGconn *conn)
* If we sent the COPY command in extended-query mode, we must issue a
* Sync as well.
*/
- if (conn->queryclass != PGQUERY_SIMPLE)
+ if (conn->cmd_queue_head &&
+ conn->cmd_queue_head->queryclass != PGQUERY_SIMPLE)
{
if (pqPutMsgStart('S', conn) < 0 ||
pqPutMsgEnd(conn) < 0)
@@ -1897,6 +1949,9 @@ pqFunctionCall3(PGconn *conn, Oid fnid,
int avail;
int i;
+ /* already validated by PQfn */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
/* PQfn already validated connection state */
if (pqPutMsgStart('F', conn) < 0 || /* function call msg */
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index fa9b62a844..cee42d4843 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQpipelineSync(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 2f052f61f8..6374ec657a 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,21 +217,16 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
} PGAsyncStatusType;
-/* PGQueryClass tracks which query protocol we are now executing */
-typedef enum
-{
- PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
- PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
- PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
-} PGQueryClass;
-
/* Target server type (decoded value of target_session_attrs) */
typedef enum
{
@@ -305,6 +300,29 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/*
+ * PGQueryClass tracks which query protocol is in use for each command queue
+ * entry, or special operation in execution
+ */
+typedef enum
+{
+ PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
+ PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
+ PGQUERY_PREPARE, /* Parse only (PQprepare) */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync (at end of a pipeline) */
+} PGQueryClass;
+
+/*
+ * An entry in the pending command queue.
+ */
+typedef struct PGcmdQueueEntry
+{
+ PGQueryClass queryclass; /* Query type */
+ char *query; /* SQL command, or NULL if none/unknown/OOM */
+ struct PGcmdQueueEntry *next; /* list link */
+} PGcmdQueueEntry;
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -389,12 +407,11 @@ struct pg_conn
ConnStatusType status;
PGAsyncStatusType asyncStatus;
PGTransactionStatusType xactStatus; /* never changes to ACTIVE */
- PGQueryClass queryclass;
- char *last_query; /* last SQL command, or NULL if unknown */
char last_sqlstate[6]; /* last reported SQLSTATE */
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -407,6 +424,19 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The pending command queue as a singly-linked list. Head is the command
+ * currently in execution, tail is where new commands are added.
+ */
+ PGcmdQueueEntry *cmd_queue_head;
+ PGcmdQueueEntry *cmd_queue_tail;
+
+ /*
+ * To save malloc traffic, we don't free entries right away; instead we
+ * save them in this list for possible reuse.
+ */
+ PGcmdQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -622,6 +652,7 @@ extern void pqSaveMessageField(PGresult *res, char code,
extern void pqSaveParameterStatus(PGconn *conn, const char *name,
const char *value);
extern int pqRowProcessor(PGconn *conn, const char **errmsgp);
+extern void pqCommandQueueAdvance(PGconn *conn);
extern int PQsendQueryContinue(PGconn *conn, const char *query);
/* === in fe-protocol3.c === */
@@ -795,6 +826,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold, for pipeline mode.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..93e7829c67 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -10,6 +10,7 @@ SUBDIRS = \
delay_execution \
dummy_index_am \
dummy_seclabel \
+ libpq_pipeline \
plsample \
snapshot_too_old \
test_bloomfilter \
diff --git a/src/test/modules/libpq_pipeline/.gitignore b/src/test/modules/libpq_pipeline/.gitignore
new file mode 100644
index 0000000000..3a11e786b8
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/libpq_pipeline
diff --git a/src/test/modules/libpq_pipeline/Makefile b/src/test/modules/libpq_pipeline/Makefile
new file mode 100644
index 0000000000..b798f5fbbc
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/libpq_pipeline/Makefile
+
+PROGRAM = libpq_pipeline
+OBJS = libpq_pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS_INTERNAL += $(libpq_pgport)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/libpq_pipeline
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/libpq_pipeline/README b/src/test/modules/libpq_pipeline/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/libpq_pipeline/libpq_pipeline.c b/src/test/modules/libpq_pipeline/libpq_pipeline.c
new file mode 100644
index 0000000000..03eb3df504
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/libpq_pipeline.c
@@ -0,0 +1,1303 @@
+/*-------------------------------------------------------------------------
+ *
+ * libpq_pipeline.c
+ * Verify libpq pipeline execution functionality
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/test/modules/libpq_pipeline/libpq_pipeline.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+#ifdef HAVE_SYS_SELECT_H
+#include <sys/select.h>
+#endif
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "libpq_pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+
+ fflush(stdout);
+
+ fprintf(stderr, "\n%s:%d: ", progname, line);
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ Assert(fmt[strlen(fmt) - 1] != '\n');
+ fprintf(stderr, "\n");
+ exit(1);
+}
+
+static void
+test_disallowed_in_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("Pipeline sync failed: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ PQclear(res);
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_pipeline_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+ bool goterror;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s", PQerrorMessage(conn));
+
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* Try to send two queries in one command */
+ if (PQsendQuery(conn, "SELECT 1; SELECT 2") != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ goterror = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_FATAL_ERROR:
+ if (strcmp(PQresultErrorField(res, PG_DIAG_SQLSTATE), "42601") != 0)
+ pg_fatal("expected error about multiple commands, got %s",
+ PQerrorMessage(conn));
+ printf("got expected %s", PQerrorMessage(conn));
+ goterror = true;
+ break;
+ default:
+ pg_fatal("got unexpected status %s", PQresStatus(PQresultStatus(res)));
+ break;
+ }
+ }
+ if (!goterror)
+ pg_fatal("did not get cannot-insert-multiple-commands error");
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("got NULL result");
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Test single-row mode with an error partways */
+ if (PQsendQuery(conn, "SELECT 1.0/g FROM generate_series(3, -1, -1) g") != 1)
+ pg_fatal("failed to send query: %s", PQerrorMessage(conn));
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ PQsetSingleRowMode(conn);
+ goterror = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_SINGLE_TUPLE:
+ printf("got row: %s\n", PQgetvalue(res, 0, 0));
+ break;
+ case PGRES_FATAL_ERROR:
+ if (strcmp(PQresultErrorField(res, PG_DIAG_SQLSTATE), "22012") != 0)
+ pg_fatal("expected division-by-zero, got: %s (%s)",
+ PQerrorMessage(conn),
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ printf("got expected division-by-zero\n");
+ goterror = true;
+ break;
+ default:
+ pg_fatal("got unexpected result %s", PQresStatus(PQresultStatus(res)));
+ }
+ PQclear(res);
+ }
+ if (!goterror)
+ pg_fatal("did not get division-by-zero error");
+ /* the third pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from third pipeline sync",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+};
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ enum PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's output buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step++;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step++;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQpipelineSync(conn) == 1)
+ {
+ fprintf(stdout, "pipeline sync sent\n");
+ send_step++;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: pipeline sync failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_prepared(PGconn *conn)
+{
+ PGresult *res = NULL;
+ Oid param_oids[1] = {INT4OID};
+ Oid expected_oids[4];
+ Oid typ;
+
+ fprintf(stderr, "prepared... ");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "select_one", "SELECT $1, '42', $1::numeric, "
+ "interval '1 sec'",
+ 1, param_oids) != 1)
+ pg_fatal("preparing query failed: %s", PQerrorMessage(conn));
+ expected_oids[0] = INT4OID;
+ expected_oids[1] = TEXTOID;
+ expected_oids[2] = NUMERICOID;
+ expected_oids[3] = INTERVALOID;
+ if (PQsendDescribePrepared(conn, "select_one") != 1)
+ pg_fatal("failed to send describePrepared: %s", PQerrorMessage(conn));
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("expected COMMAND_OK, got %s", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("expected NULL result");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned NULL");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("expected COMMAND_OK, got %s", PQresStatus(PQresultStatus(res)));
+ if (PQnfields(res) != lengthof(expected_oids))
+ pg_fatal("expected %d columns, got %d",
+ lengthof(expected_oids), PQnfields(res));
+ for (int i = 0; i < PQnfields(res); i++)
+ {
+ typ = PQftype(res, i);
+ if (typ != expected_oids[i])
+ pg_fatal("field %d: expected type %u, got %u",
+ i, expected_oids[i], typ);
+ }
+ PQclear(res);
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("expected NULL result");
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("expected PGRES_PIPELINE_SYNC, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("could not exit pipeline mode: %s", PQerrorMessage(conn));
+
+ PQexec(conn, "BEGIN");
+ PQexec(conn, "DECLARE cursor_one CURSOR FOR SELECT 1");
+ PQenterPipelineMode(conn);
+ if (PQsendDescribePortal(conn, "cursor_one") != 1)
+ pg_fatal("PQsendDescribePortal failed: %s", PQerrorMessage(conn));
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("expected NULL result");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("expected COMMAND_OK, got %s", PQresStatus(PQresultStatus(res)));
+
+ typ = PQftype(res, 0);
+ if (typ != INT4OID)
+ pg_fatal("portal: expected type %u, got %u",
+ INT4OID, typ);
+ PQclear(res);
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("expected NULL result");
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("expected PGRES_PIPELINE_SYNC, got %s", PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("could not exit pipeline mode: %s", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our output buffer will give us enough slush
+ * to work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded");
+
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ isSingleTuple = true;
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ PGresult *res;
+ bool expect_null;
+ int num_syncs = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s",
+ PQerrorMessage(conn));
+ if (PQsendPrepare(conn, "rollback", "ROLLBACK", 0, NULL) != 1)
+ pg_fatal("could not send prepare on pipeline: %s",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * send a ROLLBACK using a prepared stmt. Doesn't work because we need to
+ * get out of the pipeline-aborted state first.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /* This insert fails because we're in pipeline-aborted state */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ num_syncs++;
+
+ /*
+ * This insert fails even though the pipeline got a SYNC, because we're in
+ * an aborted transaction
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ num_syncs++;
+
+ /*
+ * Send ROLLBACK using prepared stmt. This one works because we just did
+ * PQpipelineSync above.
+ */
+ if (PQsendQueryPrepared(conn, "rollback", 0, NULL, NULL, NULL, 1) != 1)
+ pg_fatal("failed to execute prepared: %s",
+ PQerrorMessage(conn));
+
+ /*
+ * Now that we're out of a transaction and in pipeline-good mode, this
+ * insert works
+ */
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s",
+ PQerrorMessage(conn));
+ /* Send two syncs now -- match up to SYNC messages below */
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ num_syncs++;
+ if (PQpipelineSync(conn) != 1)
+ pg_fatal("pipeline sync failed: %s", PQerrorMessage(conn));
+ num_syncs++;
+
+ expect_null = false;
+ for (int i = 0;; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ printf("%d: got NULL result\n", i);
+ if (!expect_null)
+ pg_fatal("did not expect NULL here");
+ expect_null = false;
+ continue;
+ }
+ restype = PQresultStatus(res);
+ printf("%d: got status %s", i, PQresStatus(restype));
+ if (expect_null)
+ pg_fatal("expected NULL");
+ if (restype == PGRES_FATAL_ERROR)
+ printf("; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ printf(": command didn't run because pipeline aborted\n");
+ }
+ else
+ printf("\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_syncs--;
+ else
+ expect_null = true;
+ if (num_syncs <= 0)
+ break;
+ }
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("returned something extra after all the syncs: %s",
+ PQresStatus(PQresultStatus(res)));
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s", PQerrorMessage(conn));
+
+ /* We expect to find one tuple containing the value "3" */
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("failed to obtain result: %s", PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("did not get 1 tuple");
+ if (strcmp(PQgetvalue(res, 0, 0), "3") != 0)
+ pg_fatal("did not get expected tuple");
+ PQclear(res);
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s tests", progname);
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+}
+
+static void
+print_test_list(void)
+{
+ printf("disallowed_in_pipeline\n");
+ printf("multi_pipelines\n");
+ printf("pipeline_abort\n");
+ printf("pipelined_insert\n");
+ printf("prepared\n");
+ printf("simple_pipeline\n");
+ printf("singlerow\n");
+ printf("transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+ PGresult *res;
+
+ if (strcmp(argv[1], "tests") == 0)
+ {
+ print_test_list();
+ exit(0);
+ }
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+ res = PQexec(conn, "SET lc_messages TO \"C\"");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to set lc_messages: %s", PQerrorMessage(conn));
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed_in_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipelines") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_pipeline_abort(conn);
+ else if (strcmp(argv[1], "pipelined_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "prepared") == 0)
+ test_prepared(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
new file mode 100644
index 0000000000..ba15b64ca7
--- /dev/null
+++ b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl
@@ -0,0 +1,28 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 8;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+
+my ($out, $err) = run_command(['libpq_pipeline', 'tests']);
+die "oops: $err" unless $err eq '';
+my @tests = split(/\s/, $out);
+
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'libpq_pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "libpq_pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 74fde40e3a..a184404e21 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -33,10 +33,11 @@ my @unlink_on_exit;
# Set of variables for modules in contrib/ and src/test/modules/
my $contrib_defines = { 'refint' => 'REFINT_VERBOSE' };
-my @contrib_uselibpq = ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo');
-my @contrib_uselibpgport = ('oid2name', 'vacuumlo');
-my @contrib_uselibpgcommon = ('oid2name', 'vacuumlo');
-my $contrib_extralibs = undef;
+my @contrib_uselibpq =
+ ('dblink', 'oid2name', 'postgres_fdw', 'vacuumlo', 'libpq_pipeline');
+my @contrib_uselibpgport = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my @contrib_uselibpgcommon = ('libpq_pipeline', 'oid2name', 'vacuumlo');
+my $contrib_extralibs = { 'libpq_pipeline' => ['ws2_32.lib'] };
my $contrib_extraincludes = { 'dblink' => ['src/backend'] };
my $contrib_extrasource = {
'cube' => [ 'contrib/cube/cubescan.l', 'contrib/cube/cubeparse.y' ],
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 61cf4eae1f..9e6777e9d0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1563,10 +1563,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcmdQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
--
2.20.1
v37-0002-Add-libpq-pipeline-mode-support-to-pgbench.patch (text/x-diff, 9.2K) download
From 2c50bf03d46fe2767c271dafb22c20e8ead5e510 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 15 Mar 2021 15:07:59 -0300
Subject: [PATCH v37 2/2] Add libpq pipeline mode support to pgbench
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Author: Daniel Vérité <daniel@manitou-mail.org>
Discussion: https://postgr.es/m/b4e34135-2bd9-4b8a-94ca-27d760da26d7@manitou-mail.org
---
src/bin/pgbench/pgbench.c | 128 +++++++++++++++++--
src/bin/pgbench/t/001_pgbench_with_server.pl | 61 +++++++++
2 files changed, 176 insertions(+), 13 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6a214669c..ba7b35d83c 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -395,10 +395,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -530,7 +531,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2568,6 +2571,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2757,11 +2764,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2805,8 +2826,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2874,6 +2897,13 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode: %s", st->id,
+ PQerrorMessage(st->con));
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3127,13 +3157,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3273,7 +3326,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3516,6 +3577,45 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQpipelineSync(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send a pipeline sync");
+ return CSTATE_ABORTED;
+ }
+ /* Now wait for the PGRES_PIPELINE_SYNC and exit pipeline mode there */
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4725,7 +4825,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index daffc18e52..d07b36faa7 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -755,6 +755,67 @@ pgbench(
}
});
+# Working \startpipeline
+pgbench(
+ '-t 1 -n -M extended',
+ 0,
+ [ qr{type: .*/001_pgbench_pipeline}, qr{processed: 1/1} ],
+ [],
+ 'pgbench startpipeline command',
+ {
+ '001_pgbench_pipeline' => q{
+-- test startpipeline
+\startpipeline
+} . "select 1;\n" x 10 . q{
+\endpipeline
+}
+ });
+
+# Try \startpipeline twice
+pgbench(
+ '-t 1 -n -M extended',
+ 2,
+ [],
+ [qr{already in pipeline mode}],
+ 'pgbench startpipeline command',
+ {
+ '001_pgbench_pipeline_2' => q{
+-- startpipeline twice
+\startpipeline
+\startpipeline
+}
+ });
+
+# Try to end a pipeline that hasn't started
+pgbench(
+ '-t 1 -n -M extended',
+ 2,
+ [],
+ [qr{not in pipeline mode}],
+ 'pgbench startpipeline command',
+ {
+ '001_pgbench_pipeline_3' => q{
+-- pipeline not started
+\endpipeline
+}
+ });
+
+# Try \gset in pipeline mode
+pgbench(
+ '-t 1 -n -M extended',
+ 2,
+ [],
+ [qr{gset is not allowed in pipeline mode}],
+ 'pgbench startpipeline command',
+ {
+ '001_pgbench_pipeline_gset' => q{
+\startpipeline
+select 1 \gset f
+\endpipeline
+}
+ });
+
+
# trigger many expression errors
my @errors = (
--
2.20.1
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 20:51 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 21:40 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-06 00:35 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-11 16:00 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-11 18:49 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Tom Lane <tgl@sss.pgh.pa.us>
2021-03-13 04:28 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-15 18:54 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-15 19:00 ` Justin Pryzby <pryzby@telsasoft.com>
2021-03-15 19:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Justin Pryzby @ 2021-03-15 19:00 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Álvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Aya Iwata <iwata.aya@jp.fujitsu.com>; Daniel Vérité <daniel@manitou-mail.org>; David G. Johnston <david.g.johnston@gmail.com>; Kirk Jamison <k.jamison@fujitsu.com>; Michael Paquier <michael.paquier@gmail.com>; Nikhil Sontakke <nikhils@2ndquadrant.com>; Vaishnavi Prabakaran <VaishnaviP@fast.au.fujitsu.com>
Are you going to update the assertion ?
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 20:51 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 21:40 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-06 00:35 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-11 16:00 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-11 18:49 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Tom Lane <tgl@sss.pgh.pa.us>
2021-03-13 04:28 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-15 18:54 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-15 19:00 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
@ 2021-03-15 19:47 ` Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 0 replies; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-15 19:47 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Andres Freund <andres@anarazel.de>; Aya Iwata <iwata.aya@jp.fujitsu.com>; Daniel Vérité <daniel@manitou-mail.org>; David G. Johnston <david.g.johnston@gmail.com>; Kirk Jamison <k.jamison@fujitsu.com>; Michael Paquier <michael.paquier@gmail.com>; Nikhil Sontakke <nikhils@2ndquadrant.com>; Vaishnavi Prabakaran <VaishnaviP@fast.au.fujitsu.com>
On 2021-Mar-15, Justin Pryzby wrote:
> Are you going to update the assertion ?
>
> +#if 0
> Assert((meta == META_NONE && varprefix == NULL) ||
> ((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
> +#endif
Yeah, caught that just after sending. Here's a notpatch.
--
Álvaro Herrera 39°49'30"S 73°17'W
"La virtud es el justo medio entre dos defectos" (Aristóteles)
Attachments:
fixassert.notpatch (text/plain, 726B) download
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ba7b35d83c..e69d43b26b 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -2823,13 +2823,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
int qrynum = 0;
/*
- * varprefix should be set only with \gset or \aset, and SQL commands do
- * not need it.
+ * varprefix should be set only with \gset or \aset, and \endpipeline and
+ * SQL commands do not need it.
*/
-#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
+ ((meta == META_ENDPIPELINE) && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
-#endif
res = PQgetResult(st->con);
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 20:51 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 21:40 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-06 00:35 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2021-03-16 20:00 ` Andres Freund <andres@anarazel.de>
2021-03-18 11:28 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
1 sibling, 1 reply; 182+ messages in thread
From: Andres Freund @ 2021-03-16 20:00 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Zhihong Yu <zyu@yugabyte.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>
Hi,
On 2021-03-05 21:35:59 -0300, Alvaro Herrera wrote:
> I'll take the weekend to think about the issue with conn->last_query and
> conn->queryclass that I mentioned yesterday; other than that detail my
> feeling is that this is committable, so I'll be looking at getting this
> pushed early next weeks, barring opinions from others.
It is *very* exciting to see this being merged. Thanks for all the work
to all that contributed!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: [HACKERS] PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:21 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-10-02 09:31 ` Re: PATCH: Batch/pipelining support for libpq Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:47 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 20:51 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 21:40 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-06 00:35 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-16 20:00 ` Re: [HACKERS] PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
@ 2021-03-18 11:28 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
0 siblings, 0 replies; 182+ messages in thread
From: Matthieu Garrigues @ 2021-03-18 11:28 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Justin Pryzby <pryzby@telsasoft.com>; pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>; Zhihong Yu <zyu@yugabyte.com>; Ibrar Ahmed <ibrar.ahmad@gmail.com>
Thanks a lot for the merge. I did some tests and the master branch
runs up to 15% faster than the last
patch I tried (v22). Amazing!
Cheers,
Matthieu Garrigues
On Tue, Mar 16, 2021 at 9:00 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2021-03-05 21:35:59 -0300, Alvaro Herrera wrote:
> > I'll take the weekend to think about the issue with conn->last_query and
> > conn->queryclass that I mentioned yesterday; other than that detail my
> > feeling is that this is committable, so I'll be looking at getting this
> > pushed early next weeks, barring opinions from others.
>
> It is *very* exciting to see this being merged. Thanks for all the work
> to all that contributed!
>
> Greetings,
>
> Andres Freund
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 02:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-06-03 11:51 ` Re: PATCH: Batch/pipelining support for libpq Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-18 06:04 ` Re: PATCH: Batch/pipelining support for libpq Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Re: PATCH: Batch/pipelining support for libpq Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2017-02-17 05:17 ` Re: PATCH: Batch/pipelining support for libpq Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-07 16:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-08 02:04 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-10 13:52 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 17:19 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-27 15:27 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` Re: PATCH: Batch/pipelining support for libpq David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Re: PATCH: Batch/pipelining support for libpq Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-20 15:51 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 00:29 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2017-06-22 11:23 ` Daniel Verite <daniel@manitou-mail.org>
1 sibling, 0 replies; 182+ messages in thread
From: Daniel Verite @ 2017-06-22 11:23 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Andres Freund <andres@anarazel.de>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Michael Paquier <michael.paquier@gmail.com>; David Steele <david@pgmasters.net>; Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>; Haribabu Kommi <kommi.haribabu@gmail.com>; Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>; Dmitry Igrishin <dmitigr@gmail.com>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>; Iwata, Aya <iwata.aya@jp.fujitsu.com>
Craig Ringer wrote:
> The kernel will usually do some packet aggregation unless we use
> TCP_NODELAY (which we don't and shouldn't)
Not sure. As a point of comparison, Oracle has it as a tunable
parameter (TCP.NODELAY), and they changed its default from
No to Yes starting from their 10g R2.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
@ 2016-05-24 04:02 ` Craig Ringer <craig@2ndquadrant.com>
2016-05-24 04:28 ` Re: PATCH: Batch/pipelining support for libpq Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2016-05-24 04:02 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 24 May 2016 at 00:00, Michael Paquier <michael.paquier@gmail.com> wrote:
>
> Did you consider the use of simple_list.c instead of introducing a new
> mimic as PGcommandQueueEntry? It would be cool avoiding adding new
> list emulations on frontends.
>
I'd have to extend simple_list to add a generic object version, like
struct my_list_elem
{
PG_SIMPLE_LIST_ATTRS;
mytype mycol;
myothertype myothercol;
}
Objections?
I could add a void* version that's a simple clone of the string version,
but having to malloc both a list cell and its contents separately is
annoying.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael.paquier@gmail.com>
2016-05-24 04:02 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-05-24 04:28 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 182+ messages in thread
From: Tom Lane @ 2016-05-24 04:28 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
Craig Ringer <craig@2ndquadrant.com> writes:
> On 24 May 2016 at 00:00, Michael Paquier <michael.paquier@gmail.com> wrote:
>> Did you consider the use of simple_list.c instead of introducing a new
>> mimic as PGcommandQueueEntry? It would be cool avoiding adding new
>> list emulations on frontends.
> I'd have to extend simple_list to add a generic object version, like
> struct my_list_elem
> {
> PG_SIMPLE_LIST_ATTRS;
> mytype mycol;
> myothertype myothercol;
> }
> Objections?
That doesn't look exactly "generic".
> I could add a void* version that's a simple clone of the string version,
> but having to malloc both a list cell and its contents separately is
> annoying.
I'd be okay with a void* version, but I'm not sure about this.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-05-24 13:55 ` Jim Nasby <Jim.Nasby@BlueTreble.com>
2 siblings, 0 replies; 182+ messages in thread
From: Jim Nasby @ 2016-05-24 13:55 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; pgsql-hackers; +Cc: Michael Paquier <michael.paquier@gmail.com>; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 5/23/16 4:19 AM, Craig Ringer wrote:
> + Batching less useful when information from one operation is required by the
SB "Batching is less useful".
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532) mobile: 512-569-9461
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-10-12 11:51 ` Shay Rojansky <roji@roji.org>
2016-10-14 05:02 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2 siblings, 1 reply; 182+ messages in thread
From: Shay Rojansky @ 2016-10-12 11:51 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: pgsql-hackers; Michael Paquier <michael.paquier@gmail.com>; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
Hi all. I thought I'd share some experience from Npgsql regarding
batching/pipelining - hope this isn't off-topic.
Npgsql has supported batching for quite a while, similar to what this patch
proposes - with a single Sync message is sent at the end.
It has recently come to my attention that this implementation is
problematic because it forces the batch to occur within a transaction; in
other words, there's no option for a non-transactional batch. This can be a
problem for several reasons: users may want to sent off a batch of inserts,
not caring whether one of them fails (e.g. because of a unique constraint
violation). In other words, in some scenarios it may be appropriate for
later batched statements to be executed when an earlier batched statement
raised an error. If Sync is only sent at the very end, this isn't possible.
Another example of a problem (which actually happened) is that transactions
acquire row-level locks, and so may trigger deadlocks if two different
batches update the same rows in reverse order. Both of these issues
wouldn't occur if the batch weren't implicitly batched.
My current plan is to modify the batch implementation based on whether
we're in an (explicit) transaction or not. If we're in a transaction, then
it makes perfect sense to send a single Sync at the end as is being
proposed here - any failure would cause the transaction to fail anyway, so
skipping all subsequent statements until the batch's end makes sense.
However, if we're not in an explicit transaction, I plan to insert a Sync
message after each individual Execute, making non-transactional batched
statements more or less identical in behavior to non-transactional
unbatched statements. Note that this mean that a batch can generate
multiple errors, not just one.
I'm sharing this since it may be relevant to the libpq batching
implementation as well, and also to get any feedback regarding how Npgsql
should act.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-12 11:51 ` Re: PATCH: Batch/pipelining support for libpq Shay Rojansky <roji@roji.org>
@ 2016-10-14 05:02 ` Craig Ringer <craig@2ndquadrant.com>
2016-10-14 10:09 ` Re: PATCH: Batch/pipelining support for libpq Shay Rojansky <roji@roji.org>
0 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2016-10-14 05:02 UTC (permalink / raw)
To: Shay Rojansky <roji@roji.org>; +Cc: pgsql-hackers; Michael Paquier <michael.paquier@gmail.com>; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 12 October 2016 at 19:51, Shay Rojansky <roji@roji.org> wrote:
> Hi all. I thought I'd share some experience from Npgsql regarding
> batching/pipelining - hope this isn't off-topic.
Not at all.
> Npgsql has supported batching for quite a while, similar to what this patch
> proposes - with a single Sync message is sent at the end.
PgJDBC does too. The benefits of it there are what prompted me to do
this, not only so libpq users can use it directly but so psqlODBC,
psycopg2, etc can benefit from it if they choose to expose it.
> It has recently come to my attention that this implementation is problematic
> because it forces the batch to occur within a transaction; in other words,
> there's no option for a non-transactional batch.
That's not strictly the case. If you explicitly BEGIN and COMMIT,
those operations are honoured within the batch.
What's missing is implicit transactions. Usually if you send a series
of messages they will each get their own implicit transaction. If you
batch them, the whole lot gets an implicit transaction. This is
because the PostgreSQL v3 protocol conflates transaction delineation
with protocol error recovery into a single Sync message type.
It's very similar to the behaviour of multi-statements, where:
psql -c "CREATE TABLE fred(x integer); DROP TABLE notexists;"
doesn't create "fred", but
psql -c "BEGIN; CREATE TABLE fred(x integer); COMMIT; BEGIN; DROP
TABLE notexists; COMMMIT;"
... does. And in fact it's for almost the same reason. They're sent as
a single SimpleQuery message by psql and split up client side, but the
effect is the same as two separate Query messages followed by a Sync.
It isn't simple to manage this client-side, because libpq doesn't know
whether a given command string may contain transaction delimiting
statements or not and can't reliably look for them without client-side
parsing of the SQL. So it can't dispatch its own BEGIN/COMMIT around
statements in a batch that it thinks might be intended to be
autocommit, and anyway that'd result in sending 3 queries for every 1
client query, which would suck.
If the mythical v4 protocol ever happens I'd want to split Sync into
two messages, one which is a protocol synchronisation message and
another that is a transaction delimiter. Or give it flags or whatever.
In the mean time:
> This can be a problem for
> several reasons: users may want to sent off a batch of inserts, not caring
> whether one of them fails (e.g. because of a unique constraint violation).
> In other words, in some scenarios it may be appropriate for later batched
> statements to be executed when an earlier batched statement raised an error.
> If Sync is only sent at the very end, this isn't possible.
Right, and that remains the case even with explicit transaction
delineation, because the first ERROR causes processing of all
subsequent messages to be skipped.
The design I have in libpq allows for this by allowing the client to
delimit batches without ending batch mode, concurrently consuming a
stream of multiple batches. Each endbatch is a Sync. So a client that
wants autocommit-like behavour can send a series of 1-query batches.
I think I'll need to document this a bit better since it's more subtle
than I properly explained.
> Another example
> of a problem (which actually happened) is that transactions acquire
> row-level locks, and so may trigger deadlocks if two different batches
> update the same rows in reverse order. Both of these issues wouldn't occur
> if the batch weren't implicitly batched.
Same solution as above.
> My current plan is to modify the batch implementation based on whether we're
> in an (explicit) transaction or not. If we're in a transaction, then it
> makes perfect sense to send a single Sync at the end as is being proposed
> here - any failure would cause the transaction to fail anyway, so skipping
> all subsequent statements until the batch's end makes sense. However, if
> we're not in an explicit transaction, I plan to insert a Sync message after
> each individual Execute, making non-transactional batched statements more or
> less identical in behavior to non-transactional unbatched statements. Note
> that this mean that a batch can generate multiple errors, not just one.
Yes, that's what I suggest, and basically what the libpq batch
interface does, though it expects the client to deal with the
transaction boundaries.
You will need to think hard about transaction boundaries as they
relate to multi-statements unless nPgSQL parses out each statement
from multi-statement strings like PgJDBC does. Otherwise a user can
sneak in:
somestatement; BEGIN; someotherstatement;
or
somestatement; CoMMiT; otherstatement;
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-12 11:51 ` Re: PATCH: Batch/pipelining support for libpq Shay Rojansky <roji@roji.org>
2016-10-14 05:02 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-10-14 10:09 ` Shay Rojansky <roji@roji.org>
2016-10-14 12:53 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Shay Rojansky @ 2016-10-14 10:09 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: pgsql-hackers; Michael Paquier <michael.paquier@gmail.com>; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
>
> > It has recently come to my attention that this implementation is
> problematic
> > because it forces the batch to occur within a transaction; in other
> words,
> > there's no option for a non-transactional batch.
>
> That's not strictly the case. If you explicitly BEGIN and COMMIT,
> those operations are honoured within the batch.
>
I wasn't precise in my formulation (although I think we understand each
other)... The problem I'm trying to address here is the fact that in the
"usual" batching implementation (i.e. where a single Sync message is sent
at the end of the batch), there's no support for batches which have no
transactions whatsoever (i.e. where each statement is auto-committed and
errors in earlier statements don't trigger skipping of later statements).
What's missing is implicit transactions. Usually if you send a series
> of messages they will each get their own implicit transaction. If you
> batch them, the whole lot gets an implicit transaction. This is
> because the PostgreSQL v3 protocol conflates transaction delineation
> with protocol error recovery into a single Sync message type.
>
That's right. I sent a message complaining about this conflation a while
ago:
https://www.postgresql.org/message-id/CADT4RqDdo9EcFbxwB_YO2H3BVZ0t-1qqZ%3D%2B%2BdVMnYaN6BpyUGQ%40ma....
There weren't any responses, although I'll add a note on the wiki on this
as a requested feature for the v4 protocol.
If the mythical v4 protocol ever happens I'd want to split Sync into
> two messages, one which is a protocol synchronisation message and
> another that is a transaction delimiter. Or give it flags or whatever.
>
Totally agree.
> In the mean time:
>
> > This can be a problem for
> > several reasons: users may want to sent off a batch of inserts, not
> caring
> > whether one of them fails (e.g. because of a unique constraint
> violation).
> > In other words, in some scenarios it may be appropriate for later batched
> > statements to be executed when an earlier batched statement raised an
> error.
> > If Sync is only sent at the very end, this isn't possible.
>
> Right, and that remains the case even with explicit transaction
> delineation, because the first ERROR causes processing of all
> subsequent messages to be skipped.
>
> The design I have in libpq allows for this by allowing the client to
> delimit batches without ending batch mode, concurrently consuming a
> stream of multiple batches. Each endbatch is a Sync. So a client that
> wants autocommit-like behavour can send a series of 1-query batches.
> I think I'll need to document this a bit better since it's more subtle
> than I properly explained.
>
Ah, I see. libpq's API is considerably more low-level than what Npgsql
needs to provide. If I understand correctly, you allow users to specify
exactly where to insert Sync messages (if at all), so that any number of
statements arbitrarily interleaved with Sync messages can be sent without
starting to read any results. If so, then the user indeed has everything
they need to control the exact transactional behavior they want (including
full auto-commit) without compromising on performance in any way (i.e. by
increasing roundtrips).
The only minor problem I can see is that PQsendEndBatch not only adds a
Sync message to the buffer, but also flushes it. This means that you may be
forcing users to needlessly flush the buffer just because they wanted to
insert a Sync. In other words, users can't send the following messages in a
single buffer/packet:
Prepare1/Bind1/Describe1/Execute1/Sync1/Prepare2/Bind2/Describe2/Execute2/Sync2
- they have to be split into different packets. Of course, this is a
relatively minor performance issue (especially when compared to the overall
performance benefits provided by batching), and providing an API
distinction between adding a Sync and flushing the buffer may
over-complicate the API. I just thought I'd mention it.
> Yes, that's what I suggest, and basically what the libpq batch
> interface does, though it expects the client to deal with the
> transaction boundaries.
>
> You will need to think hard about transaction boundaries as they
> relate to multi-statements unless nPgSQL parses out each statement
> from multi-statement strings like PgJDBC does. Otherwise a user can
> sneak in:
>
> somestatement; BEGIN; someotherstatement;
>
> or
>
> somestatement; CoMMiT; otherstatement;
>
That's a good point. I definitely don't want to depend on client-side
parsing of SQL in any way (in fact a planned feature is to allow using
Npgsql without any sort of client-side parsing/manipulation of SQL).
However, the fact that BEGIN/COMMIT can be sent in batches doesn't appear
too problematic to me.
When it's about to send a batch, Npgsql knows whether it's in an (explicit)
transaction or not (by examining the transaction indicator on the last
ReadyForQuery message it received). If it's not in an (explicit)
transaction, it automatically inserts a Sync message after every Execute.
If some statement happens to be a BEGIN, it will be executed as a normal
statement and so on. The only issue is that if an error occurs after a
sneaked-in BEGIN, all subsequent statements will fail, and all have the
Sync messages Npgsql inserted. The end result will be a series of errors
that will be raised up to the user, but this isn't fundamentally different
from the case of a simple auto-commit batch containing multiple failures
(because of unique constraint violation or whatever) - multiple errors is
something that will have to be handled in any case.
Thanks for all your comments. Npgsql's support of batches needs to be more
complicated than libpq's since it's a more high-level interface - whereas
libpq offloads some of the sending/processing complexity to the user,
Npgsql needs to take care of most of it internally (another good example is
deadlock avoidance...).
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-12 11:51 ` Re: PATCH: Batch/pipelining support for libpq Shay Rojansky <roji@roji.org>
2016-10-14 05:02 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-14 10:09 ` Re: PATCH: Batch/pipelining support for libpq Shay Rojansky <roji@roji.org>
@ 2016-10-14 12:53 ` Craig Ringer <craig@2ndquadrant.com>
2016-10-14 14:15 ` Re: PATCH: Batch/pipelining support for libpq Shay Rojansky <roji@roji.org>
0 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2016-10-14 12:53 UTC (permalink / raw)
To: Shay Rojansky <roji@roji.org>; +Cc: pgsql-hackers; Michael Paquier <michael.paquier@gmail.com>; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 14 October 2016 at 18:09, Shay Rojansky <roji@roji.org> wrote:
>> > It has recently come to my attention that this implementation is
>> > problematic
>> > because it forces the batch to occur within a transaction; in other
>> > words,
>> > there's no option for a non-transactional batch.
>>
>> That's not strictly the case. If you explicitly BEGIN and COMMIT,
>> those operations are honoured within the batch.
>
>
> I wasn't precise in my formulation (although I think we understand each
> other)... The problem I'm trying to address here is the fact that in the
> "usual" batching implementation (i.e. where a single Sync message is sent at
> the end of the batch), there's no support for batches which have no
> transactions whatsoever (i.e. where each statement is auto-committed and
> errors in earlier statements don't trigger skipping of later statements).
Right, you can't use implicit transactions delimited by protocol
message boundaries when batching.
>> The design I have in libpq allows for this by allowing the client to
>> delimit batches without ending batch mode, concurrently consuming a
>> stream of multiple batches. Each endbatch is a Sync. So a client that
>> wants autocommit-like behavour can send a series of 1-query batches.
>
> Ah, I see. libpq's API is considerably more low-level than what Npgsql needs
> to provide. If I understand correctly, you allow users to specify exactly
> where to insert Sync messages (if at all)
They don't get total control, but they can cause a Sync to be emitted
after any given statement when in batch mode.
> so that any number of statements
> arbitrarily interleaved with Sync messages can be sent without starting to
> read any results.
Right.
> if so, then the user indeed has everything they need to
> control the exact transactional behavior they want (including full
> auto-commit) without compromising on performance in any way (i.e. by
> increasing roundtrips).
Right.
> The only minor problem I can see is that PQsendEndBatch not only adds a Sync
> message to the buffer, but also flushes it.
It only does a non-blocking flush.
> This means that you may be
> forcing users to needlessly flush the buffer just because they wanted to
> insert a Sync. In other words, users can't send the following messages in a
> single buffer/packet:
> Prepare1/Bind1/Describe1/Execute1/Sync1/Prepare2/Bind2/Describe2/Execute2/Sync2
> - they have to be split into different packets.
Oh, right. That's true, but I'm not really sure we care. I suspect
that the OS will tend to coalesce them anyway, since we're not
actually waiting until the socket sends the message. At least when the
socket isn't able to send as fast as input comes in. It might matter
for local performance in incredibly high throughput applications, but
I suspect there will be a great many other things that come first.
Anyway, the client can already control this with TCP_CORK.
> Of course, this is a
> relatively minor performance issue (especially when compared to the overall
> performance benefits provided by batching), and providing an API distinction
> between adding a Sync and flushing the buffer may over-complicate the API. I
> just thought I'd mention it.
Unnecessary IMO. If we really want to add it later we'd probably do so
by setting a "no flush on endbatch" mode and adding an explicit flush
call. But I expect TCP_CORK will satisfy all realistic needs.
> That's a good point. I definitely don't want to depend on client-side
> parsing of SQL in any way (in fact a planned feature is to allow using
> Npgsql without any sort of client-side parsing/manipulation of SQL).
> However, the fact that BEGIN/COMMIT can be sent in batches doesn't appear
> too problematic to me.
>
> When it's about to send a batch, Npgsql knows whether it's in an (explicit)
> transaction or not (by examining the transaction indicator on the last
> ReadyForQuery message it received). If it's not in an (explicit)
> transaction, it automatically inserts a Sync message after every Execute. If
> some statement happens to be a BEGIN, it will be executed as a normal
> statement and so on. The only issue is that if an error occurs after a
> sneaked-in BEGIN, all subsequent statements will fail, and all have the Sync
> messages Npgsql inserted. The end result will be a series of errors that
> will be raised up to the user, but this isn't fundamentally different from
> the case of a simple auto-commit batch containing multiple failures (because
> of unique constraint violation or whatever) - multiple errors is something
> that will have to be handled in any case.
I'm a bit hesitant about how this will interact with multi-statements
containing embedded BEGIN or COMMIT, where a single protocol message
contains multiple ; delimited queries. But at least at this time of
night I can't give a concrete problematic example.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-12 11:51 ` Re: PATCH: Batch/pipelining support for libpq Shay Rojansky <roji@roji.org>
2016-10-14 05:02 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-14 10:09 ` Re: PATCH: Batch/pipelining support for libpq Shay Rojansky <roji@roji.org>
2016-10-14 12:53 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
@ 2016-10-14 14:15 ` Shay Rojansky <roji@roji.org>
2016-10-14 14:28 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
0 siblings, 1 reply; 182+ messages in thread
From: Shay Rojansky @ 2016-10-14 14:15 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: pgsql-hackers; Michael Paquier <michael.paquier@gmail.com>; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
>
> > Of course, this is a
> > relatively minor performance issue (especially when compared to the
> overall
> > performance benefits provided by batching), and providing an API
> distinction
> > between adding a Sync and flushing the buffer may over-complicate the
> API. I
> > just thought I'd mention it.
>
> Unnecessary IMO. If we really want to add it later we'd probably do so
> by setting a "no flush on endbatch" mode and adding an explicit flush
> call. But I expect TCP_CORK will satisfy all realistic needs.
>
Unless I'm mistaken TCP_CORK is not necessarily going to work across all
platforms (e.g. Windows), although SO_LINGER (which is more standard) also
helps here.
> When it's about to send a batch, Npgsql knows whether it's in an
> (explicit)
> > transaction or not (by examining the transaction indicator on the last
> > ReadyForQuery message it received). If it's not in an (explicit)
> > transaction, it automatically inserts a Sync message after every
> Execute. If
> > some statement happens to be a BEGIN, it will be executed as a normal
> > statement and so on. The only issue is that if an error occurs after a
> > sneaked-in BEGIN, all subsequent statements will fail, and all have the
> Sync
> > messages Npgsql inserted. The end result will be a series of errors that
> > will be raised up to the user, but this isn't fundamentally different
> from
> > the case of a simple auto-commit batch containing multiple failures
> (because
> > of unique constraint violation or whatever) - multiple errors is
> something
> > that will have to be handled in any case.
>
> I'm a bit hesitant about how this will interact with multi-statements
> containing embedded BEGIN or COMMIT, where a single protocol message
> contains multiple ; delimited queries. But at least at this time of
> night I can't give a concrete problematic example.
Unless I'm mistaken, in the extended protocol you can't combine multiple ;
delimited queries in a single Parse - that's a feature supported only by
the Query message of the simple protocol. But then, if you're in the simple
protocol Sync doesn't play any role, does it? I mean, a Query message
behaves as though it's implicitly followed by a Sync message - an error in
a previous Query message doesn't cause later messages to be skipped...
Note that Npgsql only rarely uses the simple protocol for user messages.
Npgsql is a binary-only driver, and the simple protocol doesn't support
requesting binary results. So only the extended protocol is used, except
for some edge cases where it's possible to use the simple protocol for
efficiency - statements with no parameters and where the ExecuteNonQuery
API is used (i.e. the user won't access any results).
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-12 11:51 ` Re: PATCH: Batch/pipelining support for libpq Shay Rojansky <roji@roji.org>
2016-10-14 05:02 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-14 10:09 ` Re: PATCH: Batch/pipelining support for libpq Shay Rojansky <roji@roji.org>
2016-10-14 12:53 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-10-14 14:15 ` Re: PATCH: Batch/pipelining support for libpq Shay Rojansky <roji@roji.org>
@ 2016-10-14 14:28 ` Craig Ringer <craig@2ndquadrant.com>
0 siblings, 0 replies; 182+ messages in thread
From: Craig Ringer @ 2016-10-14 14:28 UTC (permalink / raw)
To: Shay Rojansky <roji@roji.org>; +Cc: pgsql-hackers; Michael Paquier <michael.paquier@gmail.com>; Manuel Kniep <m.kniep@web.de>; fujita.etsuro@lab.ntt.co.jp <fujita.etsuro@lab.ntt.co.jp>
On 14 October 2016 at 22:15, Shay Rojansky <roji@roji.org> wrote:
> Unless I'm mistaken TCP_CORK is not necessarily going to work across all
> platforms (e.g. Windows), although SO_LINGER (which is more standard) also
> helps here.
Yeah, true.
You can also twiddle TCP_NODELAY on and off on other platforms.
Anyway, my point is that it's not likely to be crucial, especially
since even without socket options the host will often do packet
combining if the queue isn't empty.
> Unless I'm mistaken, in the extended protocol you can't combine multiple ;
> delimited queries in a single Parse - that's a feature supported only by the
> Query message of the simple protocol. But then, if you're in the simple
> protocol Sync doesn't play any role, does it? I mean, a Query message
> behaves as though it's implicitly followed by a Sync message - an error in a
> previous Query message doesn't cause later messages to be skipped.
Right, good point. So that concern isn't relevant.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
@ 2020-08-31 16:05 Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 12:38 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
0 siblings, 2 replies; 182+ messages in thread
From: Matthieu Garrigues @ 2020-08-31 16:05 UTC (permalink / raw)
To: pgsql-hackers
Hi,
It seems like this patch is nearly finished. I fixed all the remaining
issues. I'm also asking
a confirmation of the test scenarios you want to see in the next
version of the patch.
> Hi,
>
> On 2020-07-10 19:01:49 -0400, Alvaro Herrera wrote:
> > Totally unasked for, here's a rebase of this patch series. I didn't do
> > anything other than rebasing to current master, solving a couple of very
> > trivial conflicts, fixing some whitespace complaints by git apply, and
> > running tests to verify everthing works.
> >
> > I don't foresee working on this at all, so if anyone is interested in
> > seeing this feature in, I encourage them to read and address
> > Horiguchi-san's feedback.
>
> Nor am I planning to do so, but I do think its a pretty important
> improvement.
>
>
Fixed
>
>
> > +/*
> > + * PQrecyclePipelinedCommand
> > + * Push a command queue entry onto the freelist. It must be a dangling entry
> > + * with null next pointer and not referenced by any other entry's next pointer.
> > + */
>
> Dangling sounds a bit like it's already freed.
>
>
Fixed
>
>
> > +/*
> > + * PQbatchSendQueue
> > + * End a batch submission by sending a protocol sync. The connection will
> > + * remain in batch mode and unavailable for new synchronous command execution
> > + * functions until all results from the batch are processed by the client.
>
> I feel like the reference to the protocol sync is a bit too low level
> for an external API. It should first document what the function does
> from a user's POV.
>
> I think it'd also be good to document whether / whether not queries can
> already have been sent before PQbatchSendQueue is called or not.
>
Fixed
>
>
>
> > + if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
> > + {
> > + /*
> > + * In an aborted batch we don't get anything from the server for each
> > + * result; we're just discarding input until we get to the next sync
> > + * from the server. The client needs to know its queries got aborted
> > + * so we create a fake PGresult to return immediately from
> > + * PQgetResult.
> > + */
> > + conn->result = PQmakeEmptyPGresult(conn,
> > + PGRES_BATCH_ABORTED);
> > + if (!conn->result)
> > + {
> > + printfPQExpBuffer(&conn->errorMessage,
> > + libpq_gettext("out of memory"));
> > + pqSaveErrorResult(conn);
> > + return 0;
>
> Is there any way an application can recover at this point? ISTM we'd be
> stuck in the previous asyncStatus, no?
>
conn->result is null when malloc(sizeof(PGresult)) returns null. It's
very unlikely unless
the server machine is out of memory, so the server will probably be
unresponsive anyway.
I'm leaving this as it is but if anyone has a solution simple to
implement I'll fix it.
>
>
> > +/* pqBatchFlush
> > + * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
> > + * In non-batch mode, data will be flushed all the time.
> > + */
> > +static int
> > +pqBatchFlush(PGconn *conn)
> > +{
> > + if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >= OUTBUFFER_THRESHOLD))
> > + return(pqFlush(conn));
> > + return 0; /* Just to keep compiler quiet */
> > +}
>
> unnecessarily long line.
>
Fixed
>
> > +/*
> > + * Connection's outbuffer threshold is set to 64k as it is safe
> > + * in Windows as per comments in pqSendSome() API.
> > + */
> > +#define OUTBUFFER_THRESHOLD 65536
>
> I don't think the comment explains much. It's fine to send more than 64k
> with pqSendSome(), they'll just be send with separate pgsecure_write()
> invocations. And only on windows.
>
> It clearly makes sense to start sending out data at a certain
> granularity to avoid needing unnecessary amounts of memory, and to make
> more efficient use of latency / serer side compute.
>
> It's not implausible that 64k is the right amount for that, I just don't
> think the explanation above is good.
>
Fixed
> > diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
> > new file mode 100644
> > index 0000000000..4d6ba266e5
> > --- /dev/null
> > +++ b/src/test/modules/test_libpq/testlibpqbatch.c
> > @@ -0,0 +1,1456 @@
> > +/*
> > + * src/test/modules/test_libpq/testlibpqbatch.c
> > + *
> > + *
> > + * testlibpqbatch.c
> > + * Test of batch execution functionality
> > + */
> > +
> > +#ifdef WIN32
> > +#include <windows.h>
> > +#endif
>
> ISTM that this shouldn't be needed in a test program like this?
> Shouldn't libpq abstract all of this away?
>
Fixed.
>
> > +static void
> > +simple_batch(PGconn *conn)
> > +{
>
> ISTM that all or at least several of these should include tests of
> transactional behaviour with pipelining (e.g. using both implicit and
> explicit transactions inside a single batch, using transactions across
> batches, multiple explicit transactions inside a batch).
>
@Andres, just to make sure I understood, here is the test scenarios I'll add:
Implicit and explicit multiple transactions:
start batch:
create_table
insert X
begin transaction
insert X
commit transaction
begin transaction
insert Y
commit transaction
end batch
Transaction across batches:
start batch:
create_table
begin transaction
insert X
end batch
start batch:
insert Y
commit transaction
end batch
>
>
> > + PGresult *res = NULL;
> > + const char *dummy_params[1] = {"1"};
> > + Oid dummy_param_oids[1] = {INT4OID};
> > +
> > + fprintf(stderr, "simple batch... ");
> > + fflush(stderr);
>
> Why do we need fflush()? IMO that shouldn't be needed in a use like
> this? Especially not on stderr, which ought to be unbuffered?
>
Removed.
>
> > + /*
> > + * Enter batch mode and dispatch a set of operations, which we'll then
> > + * process the results of as they come in.
> > + *
> > + * For a simple case we should be able to do this without interim
> > + * processing of results since our out buffer will give us enough slush to
> > + * work with and we won't block on sending. So blocking mode is fine.
> > + */
> > + if (PQisnonblocking(conn))
> > + {
> > + fprintf(stderr, "Expected blocking connection mode\n");
> > + goto fail;
> > + }
>
> Perhaps worth adding a helper for this?
>
> #define EXPECT(condition, explanation) \
>
Fixed (and saved 600 lines :).
Once I get your confirmation of the test scenarios, I'll implement
them and share another patch.
Attachments:
libpq-batch-v19.patch (application/x-patch, 86.3K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 92556c7ce0..3fc1eaa9cd 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4829,6 +4829,500 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending"/>.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function> and
+ <function>PQbatchProcessQueue</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchProcessQueue"><function>PQbatchProcessQueue</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQbatchProcessQueue</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchProcessQueue</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode"/>.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchProcessQueue(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchProcessQueue</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchProcessQueue">
+ <term>
+ <function>PQbatchProcessQueue</function>
+ <indexterm>
+ <primary>PQbatchProcessQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchProcessQueue(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4869,6 +5363,14 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <note>
+ <para>
+ On using batch mode, call <function>PQsetSingleRowMode</function>
+ immediately after a successful call of <function>PQbatchProcessQueue</function>
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index cf4653fe0f..a1b46c83ed 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 8afa5a29b4..69bf757e85 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1022,6 +1022,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..ffb18fcf47 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,8 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterBatchMode 180
+PQexitBatchMode 181
+PQbatchSendQueue 182
+PQbatchProcessQueue 183
+PQbatchStatus 184
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 7bee9dd201..25c79cb7b4 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -535,6 +535,24 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
/*
* pqDropServerData
@@ -554,6 +572,7 @@ pqDropServerData(PGconn *conn)
{
PGnotify *notify;
pgParameterStatus *pstatus;
+ PGcommandQueueEntry *queue;
/* Forget pending notifies */
notify = conn->notifyHead;
@@ -566,6 +585,14 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index eea0237c3a..b34cecd9c7 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -70,7 +72,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1210,7 +1215,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1233,6 +1238,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1331,6 +1343,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1389,31 +1405,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1461,7 +1497,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1479,20 +1588,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1516,6 +1665,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1525,6 +1678,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1637,35 +1807,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1766,6 +1943,288 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ /* error message should be set up already */
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+/*
+ * PQbatchSendQueue
+ *
+ * End a batch submission.
+ *
+ * It's legal to start submitting another batch immediately, without
+ * waiting for the results of the current batch. There's no need to end batch
+ * mode and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and
+ * including the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED
+ * state. If the whole batch is processed without error, a PGresult with
+ * PGRES_BATCH_END is produced.
+ *
+ * Queries can already have been sent before PQbatchSendQueue is called, but
+ * PQbatchSendQueue need to be called before retrieving command results.
+ *
+ * The connection will remain in batch mode and unavailable for new synchronous
+ * command execution functions until all results from the batch are processed
+ * by the client.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ return 0;
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ return 0;
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * PQbatchProcessQueue
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
/*
* PQgetResult
@@ -1842,9 +2301,30 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -2025,6 +2505,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2219,6 +2706,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2234,6 +2724,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2242,15 +2746,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2260,14 +2767,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2665,6 +3176,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3858,3 +4376,15 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 9360c541be..2ff3fa4883 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -406,6 +406,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 1696525475..9c37a9c4ff 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -217,10 +217,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -875,6 +883,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 3b6a9fbce3..9ee497433e 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -15,6 +15,10 @@
#ifndef LIBPQ_FE_H
#define LIBPQ_FE_H
+#ifdef WIN32
+#include <windows.h>
+#endif
+
#ifdef __cplusplus
extern "C"
{
@@ -97,7 +101,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -137,6 +144,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -435,6 +453,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+extern int PQbatchProcessQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1de91ae295..dee4c85382 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -394,6 +416,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -406,6 +429,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -798,6 +831,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -806,6 +844,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index a6d2ffbf9e..287214c544 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..11e8463984
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..d907063f65
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..2c361880fa
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,27 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_batch simple_batch multi_batch batch_abort timings singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'testlibpqbatch', 'dbname=postgres', "$numrows", "$testname" ],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000000..9c16999c4c
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1013 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "c.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+#define EXPECT(condition, ...) \
+ if (0 == (condition)) \
+ { \
+ fprintf(stderr, __VA_ARGS__); \
+ goto fail; \
+ }
+
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ EXPECT(!PQisnonblocking(conn), "Expected blocking connection mode\n");
+
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0),
+ "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode with work in progress should fail, but succeeded\n");
+
+ EXPECT(PQbatchSendQueue(conn), "Ending a batch failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+ EXPECT(PQbatchProcessQueue(conn), "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+
+ /* We can't PQbatchProcessQueue when there might still be pending results */
+ EXPECT(!PQbatchProcessQueue(conn), "PQbatchProcessQueue() should've failed with pending results: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from first batch item\n", PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ /* should now get an explicit sync result */
+ EXPECT(PQbatchProcessQueue(conn), "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s instead of sync result, error: %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn))
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after end batch call\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ EXPECT(!PQisnonblocking(conn), "Expected blocking connection mode: %u\n", __LINE__);
+
+ EXPECT(PQenterBatchMode(conn), "Unable to enter batch mode\n");
+
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Batch mode not activated properly\n");
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ EXPECT(PQresultStatus(res) == PGRES_FATAL_ERROR, "PQexec should fail in batch mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ EXPECT(PQsendQuery(conn, "SELECT 1") == 0, "PQsendQuery should fail in batch mode but succeeded\n");
+
+ /* Entering batch mode when already in batch mode is OK */
+ EXPECT(PQenterBatchMode(conn), "re-entering batch mode should be a no-op but failed\n");
+
+ EXPECT(!PQisBusy(conn), "PQisBusy should return false when idle in batch, returned true\n");
+
+ /* ok, back to normal command mode */
+ EXPECT(PQexitBatchMode(conn), "couldn't exit idle empty batch mode\n");
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "Batch mode not terminated properly\n");
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ EXPECT(PQexitBatchMode(conn), "batch mode exit when not in batch mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "PQexec should succeed after exiting batch mode but failed with: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the batch results */
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something in a batch before first PQbatchProcessQueue() call\n");
+
+ EXPECT(PQbatchProcessQueue(conn), "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from first batch item\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after first result before PQbatchProcessQueue() call\n");
+
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ EXPECT(PQbatchProcessQueue(conn), "PQbatchProcessQueue() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s instead of sync result, error: %s (line %u)\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ EXPECT(PQbatchProcessQueue(conn), "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from second batch item\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQbatchProcessQueue(conn), "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s from second end batch\n", PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ EXPECT(PQbatchProcessQueue(conn), "PQbatchProcessQueue() failed at first batch entry: %s\n", PQerrorMessage(conn));
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_COMMAND_OK,
+ "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ EXPECT(PQbatchProcessQueue(conn), "PQbatchProcessQueue() failed at second batch entry: %s\n", PQerrorMessage(conn));
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_FATAL_ERROR, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_ABORTED, "batch should be flagged as aborted but isn't\n");
+
+ /* third query in batch, the second insert */
+ EXPECT(PQbatchProcessQueue(conn), "PQbatchProcessQueue() failed at third batch entry: %s\n", PQerrorMessage(conn));
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_ABORTED, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n", res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_ABORTED, "batch should be flagged as aborted but isn't\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* the batch sync */
+ EXPECT(PQbatchProcessQueue(conn), "PQbatchProcessQueue() failed at first batch sync: %s\n", PQerrorMessage(conn));
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_END,
+ "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_ABORTED, "sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* the insert from the second batch */
+ EXPECT(PQbatchProcessQueue(conn), "PQbatchProcessQueue() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_COMMAND_OK, "Unexpected result code %s from first item in second batch\n", res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ EXPECT(PQbatchProcessQueue(conn), "PQbatchProcessQueue() failed at second batch sync: %s\n", PQerrorMessage(conn));
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s from second batch sync\n", res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Expected tuples, got %s: %s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ EXPECT(strcmp(val, "3") == 0, "expected only insert with value 3, got %s", val);
+ }
+
+ EXPECT(PQntuples(res) == 1, "expected 1 result, got %d", PQntuples(res));
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0), "xact start failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ EXPECT(PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0), "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ EXPECT(PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0), "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ EXPECT(PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids), "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ EXPECT(PQsetnonblocking(conn, 1) == 0, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ EXPECT(PQbatchProcessQueue(conn), "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ EXPECT(PQresultStatus(res) == status, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+
+ EXPECT(strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) == 0, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSendQueue(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsetnonblocking(conn, 0) == 0, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "prepare failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "INSERT failed: %s\n", PQerrorMessage(conn));
+
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ EXPECT(PQresultStatus(res) == PGRES_COPY_IN, "COPY: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ EXPECT(formatted < 12 + 1, "Buffer write truncated somehow\n");
+
+ EXPECT(PQputCopyData(conn, buf, formatted) == 1, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+
+ nrows--;
+ }
+
+ EXPECT(PQputCopyEnd(conn, NULL) == 1, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ instr_time start_time, end_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_pipelined(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+
+ printf("batch insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_sequential(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("sequential insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_copy(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("COPY elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQenterBatchMode(conn);
+
+ for (i=0; i < 3; i++) {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchSendQueue(conn);
+
+ i=0;
+ while (PQbatchProcessQueue(conn))
+ {
+ int isSingleTuple = 0;
+ /* Set single row mode for only first 3 SELECT queries */
+ if(i < 3)
+ {
+ r = PQsetSingleRowMode(conn);
+ if (r!=1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+ }
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ EXPECT(isSingleTuple, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ isSingleTuple=0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ }
+ i++;
+ }
+ PQexitBatchMode(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if(run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 20da7985c1..b131afaa1a 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -50,7 +50,8 @@ my @contrib_excludes = (
'pgcrypto', 'sepgsql',
'brin', 'test_extensions',
'test_misc', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'snapshot_too_old', 'unsafe_tests',
+ 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
@ 2020-09-21 12:38 ` Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:21 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 1 reply; 182+ messages in thread
From: Dave Cramer @ 2020-09-21 12:38 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; pgsql-hackers
Alvaro,
On Fri, 4 Sep 2020 at 17:26, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:
> On 2020-Aug-31, Matthieu Garrigues wrote:
>
> > It seems like this patch is nearly finished. I fixed all the remaining
> > issues. I'm also asking a confirmation of the test scenarios you want
> > to see in the next version of the patch.
>
> Hmm, apparently nobody noticed that this patch is not registered in the
> current commitfest.
>
> Since it was submitted ahead of the deadline, I'm going to add it there.
> (The patch has also undergone a lot of review already; it's certainly
> not a newcomer.)
>
I am looking for this in the commitfest and can't find it. However there is
an old commitfest entry
https://commitfest.postgresql.org/13/1024/
Do you have the link for the new one ?
Dave Cramer
www.postgres.rocks
>
>
>
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 12:38 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
@ 2020-09-21 13:21 ` Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 0 replies; 182+ messages in thread
From: Alvaro Herrera @ 2020-09-21 13:21 UTC (permalink / raw)
To: Dave Cramer <davecramer@postgres.rocks>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; pgsql-hackers
On 2020-Sep-21, Dave Cramer wrote:
Hello Dave,
> I am looking for this in the commitfest and can't find it. However there is
> an old commitfest entry
>
> https://commitfest.postgresql.org/13/1024/
>
> Do you have the link for the new one ?
Here you go:
https://commitfest.postgresql.org/29/2724/
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
@ 2020-09-21 13:08 ` Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
1 sibling, 1 reply; 182+ messages in thread
From: Dave Cramer @ 2020-09-21 13:08 UTC (permalink / raw)
To: Matthieu Garrigues <matthieu.garrigues@gmail.com>; +Cc: pgsql-hackers
Dave Cramer
www.postgres.rocks
On Mon, 31 Aug 2020 at 11:46, Matthieu Garrigues <
matthieu.garrigues@gmail.com> wrote:
> Hi,
>
> It seems like this patch is nearly finished. I fixed all the remaining
> issues. I'm also asking
> a confirmation of the test scenarios you want to see in the next
> version of the patch.
>
> > Hi,
> >
> > On 2020-07-10 19:01:49 -0400, Alvaro Herrera wrote:
> > > Totally unasked for, here's a rebase of this patch series. I didn't do
> > > anything other than rebasing to current master, solving a couple of
> very
> > > trivial conflicts, fixing some whitespace complaints by git apply, and
> > > running tests to verify everthing works.
> > >
> > > I don't foresee working on this at all, so if anyone is interested in
> > > seeing this feature in, I encourage them to read and address
> > > Horiguchi-san's feedback.
> >
> > Nor am I planning to do so, but I do think its a pretty important
> > improvement.
> >
> >
> Fixed
>
> >
> >
> > > +/*
> > > + * PQrecyclePipelinedCommand
> > > + * Push a command queue entry onto the freelist. It must be a
> dangling entry
> > > + * with null next pointer and not referenced by any other entry's
> next pointer.
> > > + */
> >
> > Dangling sounds a bit like it's already freed.
> >
> >
> Fixed
>
> >
> >
> > > +/*
> > > + * PQbatchSendQueue
> > > + * End a batch submission by sending a protocol sync. The connection
> will
> > > + * remain in batch mode and unavailable for new synchronous command
> execution
> > > + * functions until all results from the batch are processed by the
> client.
> >
> > I feel like the reference to the protocol sync is a bit too low level
> > for an external API. It should first document what the function does
> > from a user's POV.
> >
> > I think it'd also be good to document whether / whether not queries can
> > already have been sent before PQbatchSendQueue is called or not.
> >
> Fixed
>
> >
> >
> >
> > > + if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass
> != PGQUERY_SYNC)
> > > + {
> > > + /*
> > > + * In an aborted batch we don't get anything from the server for each
> > > + * result; we're just discarding input until we get to the next sync
> > > + * from the server. The client needs to know its queries got aborted
> > > + * so we create a fake PGresult to return immediately from
> > > + * PQgetResult.
> > > + */
> > > + conn->result = PQmakeEmptyPGresult(conn,
> > > + PGRES_BATCH_ABORTED);
> > > + if (!conn->result)
> > > + {
> > > + printfPQExpBuffer(&conn->errorMessage,
> > > + libpq_gettext("out of memory"));
> > > + pqSaveErrorResult(conn);
> > > + return 0;
> >
> > Is there any way an application can recover at this point? ISTM we'd be
> > stuck in the previous asyncStatus, no?
> >
>
> conn->result is null when malloc(sizeof(PGresult)) returns null. It's
> very unlikely unless
> the server machine is out of memory, so the server will probably be
> unresponsive anyway.
>
> I'm leaving this as it is but if anyone has a solution simple to
> implement I'll fix it.
>
> >
> >
> > > +/* pqBatchFlush
> > > + * In batch mode, data will be flushed only when the out buffer
> reaches the threshold value.
> > > + * In non-batch mode, data will be flushed all the time.
> > > + */
> > > +static int
> > > +pqBatchFlush(PGconn *conn)
> > > +{
> > > + if ((conn->batch_status == PQBATCH_MODE_OFF)||(conn->outCount >=
> OUTBUFFER_THRESHOLD))
> > > + return(pqFlush(conn));
> > > + return 0; /* Just to keep compiler quiet */
> > > +}
> >
> > unnecessarily long line.
> >
> Fixed
>
> >
> > > +/*
> > > + * Connection's outbuffer threshold is set to 64k as it is safe
> > > + * in Windows as per comments in pqSendSome() API.
> > > + */
> > > +#define OUTBUFFER_THRESHOLD 65536
> >
> > I don't think the comment explains much. It's fine to send more than 64k
> > with pqSendSome(), they'll just be send with separate pgsecure_write()
> > invocations. And only on windows.
> >
> > It clearly makes sense to start sending out data at a certain
> > granularity to avoid needing unnecessary amounts of memory, and to make
> > more efficient use of latency / serer side compute.
> >
> > It's not implausible that 64k is the right amount for that, I just don't
> > think the explanation above is good.
> >
>
> Fixed
>
> > > diff --git a/src/test/modules/test_libpq/testlibpqbatch.c
> b/src/test/modules/test_libpq/testlibpqbatch.c
> > > new file mode 100644
> > > index 0000000000..4d6ba266e5
> > > --- /dev/null
> > > +++ b/src/test/modules/test_libpq/testlibpqbatch.c
> > > @@ -0,0 +1,1456 @@
> > > +/*
> > > + * src/test/modules/test_libpq/testlibpqbatch.c
> > > + *
> > > + *
> > > + * testlibpqbatch.c
> > > + * Test of batch execution functionality
> > > + */
> > > +
> > > +#ifdef WIN32
> > > +#include <windows.h>
> > > +#endif
> >
> > ISTM that this shouldn't be needed in a test program like this?
> > Shouldn't libpq abstract all of this away?
> >
>
> Fixed.
>
> >
> > > +static void
> > > +simple_batch(PGconn *conn)
> > > +{
> >
> > ISTM that all or at least several of these should include tests of
> > transactional behaviour with pipelining (e.g. using both implicit and
> > explicit transactions inside a single batch, using transactions across
> > batches, multiple explicit transactions inside a batch).
> >
>
> @Andres, just to make sure I understood, here is the test scenarios I'll
> add:
>
> Implicit and explicit multiple transactions:
> start batch:
> create_table
> insert X
> begin transaction
> insert X
> commit transaction
> begin transaction
> insert Y
> commit transaction
> end batch
>
> Transaction across batches:
> start batch:
> create_table
> begin transaction
> insert X
> end batch
> start batch:
> insert Y
> commit transaction
> end batch
>
> >
> >
> > > + PGresult *res = NULL;
> > > + const char *dummy_params[1] = {"1"};
> > > + Oid dummy_param_oids[1] = {INT4OID};
> > > +
> > > + fprintf(stderr, "simple batch... ");
> > > + fflush(stderr);
> >
> > Why do we need fflush()? IMO that shouldn't be needed in a use like
> > this? Especially not on stderr, which ought to be unbuffered?
> >
> Removed.
>
> >
> > > + /*
> > > + * Enter batch mode and dispatch a set of operations, which we'll then
> > > + * process the results of as they come in.
> > > + *
> > > + * For a simple case we should be able to do this without interim
> > > + * processing of results since our out buffer will give us enough
> slush to
> > > + * work with and we won't block on sending. So blocking mode is fine.
> > > + */
> > > + if (PQisnonblocking(conn))
> > > + {
> > > + fprintf(stderr, "Expected blocking connection mode\n");
> > > + goto fail;
> > > + }
> >
> > Perhaps worth adding a helper for this?
> >
> > #define EXPECT(condition, explanation) \
> >
>
> Fixed (and saved 600 lines :).
>
>
>
> Once I get your confirmation of the test scenarios, I'll implement
> them and share another patch.
>
There was a comment upthread a while back that people should look at the
comments made in
https://www.postgresql.org/message-id/20180322.211148.187821341.horiguchi.kyotaro%40lab.ntt.co.jp
by Horiguchi-San.
From what I can tell this has not been addressed. The one big thing is the
use of PQbatchProcessQueue vs just putting it in PQgetResult.
The argument is that adding PQbatchProcessQueue is unnecessary and just
adds another step. Looking at this, it seems like putting this inside
PQgetResult would get my vote as it leaves the interface unchanged.
Dave
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
@ 2020-09-21 13:41 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
0 siblings, 1 reply; 182+ messages in thread
From: Matthieu Garrigues @ 2020-09-21 13:41 UTC (permalink / raw)
To: Dave Cramer <davecramer@postgres.rocks>; +Cc: pgsql-hackers
Matthieu Garrigues
On Mon, Sep 21, 2020 at 3:09 PM Dave Cramer <davecramer@postgres.rocks> wrote:
>>
> There was a comment upthread a while back that people should look at the comments made in https://www.postgresql.org/message-id/20180322.211148.187821341.horiguchi.kyotaro%40lab.ntt.co.jp by Horiguchi-San.
>
> From what I can tell this has not been addressed. The one big thing is the use of PQbatchProcessQueue vs just putting it in PQgetResult.
>
> The argument is that adding PQbatchProcessQueue is unnecessary and just adds another step. Looking at this, it seems like putting this inside PQgetResult would get my vote as it leaves the interface unchanged.
>
Ok. I'll merge PQbatchProcessQueue into PQgetResult. But just one
thing: I'll keep PQgetResult returning null between the result of two
batched query so the user
can know which result comes from which query.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
@ 2020-09-21 13:39 ` Dave Cramer <davecramer@postgres.rocks>
2020-09-21 14:22 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
0 siblings, 2 replies; 182+ messages in thread
From: Dave Cramer @ 2020-09-21 13:39 UTC (permalink / raw)
To: Matthieu Garrigues <matthieu.garrigues@gmail.com>; +Cc: pgsql-hackers
On Mon, 21 Sep 2020 at 09:21, Matthieu Garrigues <
matthieu.garrigues@gmail.com> wrote:
> Matthieu Garrigues
>
> On Mon, Sep 21, 2020 at 3:09 PM Dave Cramer <davecramer@postgres.rocks>
> wrote:
> >>
> > There was a comment upthread a while back that people should look at the
> comments made in
> https://www.postgresql.org/message-id/20180322.211148.187821341.horiguchi.kyotaro%40lab.ntt.co.jp
> by Horiguchi-San.
> >
> > From what I can tell this has not been addressed. The one big thing is
> the use of PQbatchProcessQueue vs just putting it in PQgetResult.
> >
> > The argument is that adding PQbatchProcessQueue is unnecessary and just
> adds another step. Looking at this, it seems like putting this inside
> PQgetResult would get my vote as it leaves the interface unchanged.
> >
>
> Ok. I'll merge PQbatchProcessQueue into PQgetResult. But just one
> thing: I'll keep PQgetResult returning null between the result of two
> batched query so the user
> can know which result comes from which query.
>
Fair enough.
There may be other things in his comments that need to be addressed. That
was the big one that stuck out for me.
Thanks for working on this!
Dave Cramer
www.postgres.rocks
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
@ 2020-09-21 14:22 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
1 sibling, 0 replies; 182+ messages in thread
From: Matthieu Garrigues @ 2020-09-21 14:22 UTC (permalink / raw)
To: Dave Cramer <davecramer@postgres.rocks>; +Cc: pgsql-hackers
On Mon, Sep 21, 2020 at 3:39 PM Dave Cramer <davecramer@postgres.rocks> wrote:
>
>
>
> On Mon, 21 Sep 2020 at 09:21, Matthieu Garrigues <matthieu.garrigues@gmail.com> wrote:
>>
>> Matthieu Garrigues
>>
>> On Mon, Sep 21, 2020 at 3:09 PM Dave Cramer <davecramer@postgres.rocks> wrote:
>> >>
>> > There was a comment upthread a while back that people should look at the comments made in https://www.postgresql.org/message-id/20180322.211148.187821341.horiguchi.kyotaro%40lab.ntt.co.jp by Horiguchi-San.
>> >
>> > From what I can tell this has not been addressed. The one big thing is the use of PQbatchProcessQueue vs just putting it in PQgetResult.
>> >
>> > The argument is that adding PQbatchProcessQueue is unnecessary and just adds another step. Looking at this, it seems like putting this inside PQgetResult would get my vote as it leaves the interface unchanged.
>> >
>>
>> Ok. I'll merge PQbatchProcessQueue into PQgetResult. But just one
>> thing: I'll keep PQgetResult returning null between the result of two
>> batched query so the user
>> can know which result comes from which query.
>
>
> Fair enough.
>
> There may be other things in his comments that need to be addressed. That was the big one that stuck out for me.
>
> Thanks for working on this!
>
Yes I already addressed the other things in the v19 patch:
https://www.postgresql.org/message-id/flat/CAJkzx4T5E-2cQe3dtv2R78dYFvz+in8PY7A8MArvLhs_pg75gg@mail....
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
@ 2020-09-21 17:55 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 182+ messages in thread
From: Matthieu Garrigues @ 2020-09-21 17:55 UTC (permalink / raw)
To: Dave Cramer <davecramer@postgres.rocks>; +Cc: pgsql-hackers
Hi Dave,
I merged PQbatchProcessQueue into PQgetResult.
One first init call to PQbatchProcessQueue was also required in
PQsendQueue to have
PQgetResult ready to read the first batch query.
Tests and documentation are updated accordingly.
Matthieu Garrigues
On Mon, Sep 21, 2020 at 3:39 PM Dave Cramer <davecramer@postgres.rocks> wrote:
>
>
>
> On Mon, 21 Sep 2020 at 09:21, Matthieu Garrigues <matthieu.garrigues@gmail.com> wrote:
>>
>> Matthieu Garrigues
>>
>> On Mon, Sep 21, 2020 at 3:09 PM Dave Cramer <davecramer@postgres.rocks> wrote:
>> >>
>> > There was a comment upthread a while back that people should look at the comments made in https://www.postgresql.org/message-id/20180322.211148.187821341.horiguchi.kyotaro%40lab.ntt.co.jp by Horiguchi-San.
>> >
>> > From what I can tell this has not been addressed. The one big thing is the use of PQbatchProcessQueue vs just putting it in PQgetResult.
>> >
>> > The argument is that adding PQbatchProcessQueue is unnecessary and just adds another step. Looking at this, it seems like putting this inside PQgetResult would get my vote as it leaves the interface unchanged.
>> >
>>
>> Ok. I'll merge PQbatchProcessQueue into PQgetResult. But just one
>> thing: I'll keep PQgetResult returning null between the result of two
>> batched query so the user
>> can know which result comes from which query.
>
>
> Fair enough.
>
> There may be other things in his comments that need to be addressed. That was the big one that stuck out for me.
>
> Thanks for working on this!
>
>
> Dave Cramer
> www.postgres.rocks
Attachments:
libpq-batch-v20.patch (text/x-patch, 84.6K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 92556c7ce0..15d0c03c89 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4829,6 +4829,465 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending"/>.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the user must call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQgetResult</function> again and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> before retrieving
+ results with <function>PQgetResult</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode"/>.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (exept that PQgetResult
+ return null to indicate that we start returning the results of next query). The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4869,6 +5328,15 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <note>
+ <para>
+ When using batch mode, activate the single row mode on the current batch query by
+ calling <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index cf4653fe0f..a1b46c83ed 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 8afa5a29b4..69bf757e85 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1022,6 +1022,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..ffb18fcf47 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,8 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterBatchMode 180
+PQexitBatchMode 181
+PQbatchSendQueue 182
+PQbatchProcessQueue 183
+PQbatchStatus 184
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 7bee9dd201..25c79cb7b4 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -535,6 +535,24 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
/*
* pqDropServerData
@@ -554,6 +572,7 @@ pqDropServerData(PGconn *conn)
{
PGnotify *notify;
pgParameterStatus *pstatus;
+ PGcommandQueueEntry *queue;
/* Forget pending notifies */
notify = conn->notifyHead;
@@ -566,6 +585,14 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index eea0237c3a..ff9a6ca253 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -70,7 +72,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1210,7 +1215,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1233,6 +1238,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1331,6 +1343,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1389,31 +1405,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1461,7 +1497,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1479,20 +1588,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1516,6 +1665,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1525,6 +1678,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1637,35 +1807,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1766,14 +1943,308 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ /* error message should be set up already */
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+
+/*
+ * internal function pqBatchProcessQueue
+ *
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+static int
+pqBatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
+/*
+ * PQbatchSendQueue
+ *
+ * End a batch submission.
+ *
+ * It's legal to start submitting another batch immediately, without
+ * waiting for the results of the current batch. There's no need to end batch
+ * mode and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and
+ * including the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED
+ * state. If the whole batch is processed without error, a PGresult with
+ * PGRES_BATCH_END is produced.
+ *
+ * Queries can already have been sent before PQbatchSendQueue is called, but
+ * PQbatchSendQueue need to be called before retrieving command results.
+ *
+ * The connection will remain in batch mode and unavailable for new synchronous
+ * command execution functions until all results from the batch are processed
+ * by the client.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ return 0;
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ return 0;
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqBatchProcessQueue so the user can call start calling getResult.
+ */
+ pqBatchProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In batch mode, once all the result of a query have been
+ * returned, PQgetResult returns NULL to let the user know that
+ * the next batched query is being processed.
+ *
+ * At the end of the batch, returns a end-of-batch result with
+ * PQresultStatus(result) == PGRES_BATCH_END.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1842,9 +2313,38 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, we prepare the processing of the results
+ * of the next query.
+ */
+ pqBatchProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -2025,6 +2525,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2219,6 +2726,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2234,6 +2744,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2242,15 +2766,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2260,14 +2787,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2665,6 +3196,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3858,3 +4396,15 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 9360c541be..2ff3fa4883 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -406,6 +406,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 1696525475..9c37a9c4ff 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -217,10 +217,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -875,6 +883,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 3b6a9fbce3..b6e3cde539 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -15,6 +15,10 @@
#ifndef LIBPQ_FE_H
#define LIBPQ_FE_H
+#ifdef WIN32
+#include <windows.h>
+#endif
+
#ifdef __cplusplus
extern "C"
{
@@ -97,7 +101,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -137,6 +144,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -435,6 +453,12 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1de91ae295..dee4c85382 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -394,6 +416,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -406,6 +429,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -798,6 +831,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -806,6 +844,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index a6d2ffbf9e..287214c544 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..11e8463984
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..d907063f65
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..2c361880fa
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,27 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_batch simple_batch multi_batch batch_abort timings singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'testlibpqbatch', 'dbname=postgres', "$numrows", "$testname" ],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000000..b82634171b
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1003 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "c.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+#define EXPECT(condition, ...) \
+ if (0 == (condition)) \
+ { \
+ fprintf(stderr, __VA_ARGS__); \
+ goto fail; \
+ }
+
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ EXPECT(!PQisnonblocking(conn), "Expected blocking connection mode\n");
+
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0),
+ "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode with work in progress should fail, but succeeded\n");
+
+ EXPECT(PQbatchSendQueue(conn), "Ending a batch failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from first batch item\n", PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when sync result PGRES_BATCH_END expected: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s instead of sync result, error: %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn))
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after end batch call\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ EXPECT(!PQisnonblocking(conn), "Expected blocking connection mode: %u\n", __LINE__);
+
+ EXPECT(PQenterBatchMode(conn), "Unable to enter batch mode\n");
+
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Batch mode not activated properly\n");
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ EXPECT(PQresultStatus(res) == PGRES_FATAL_ERROR, "PQexec should fail in batch mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ EXPECT(PQsendQuery(conn, "SELECT 1") == 0, "PQsendQuery should fail in batch mode but succeeded\n");
+
+ /* Entering batch mode when already in batch mode is OK */
+ EXPECT(PQenterBatchMode(conn), "re-entering batch mode should be a no-op but failed\n");
+
+ EXPECT(!PQisBusy(conn), "PQisBusy should return false when idle in batch, returned true\n");
+
+ /* ok, back to normal command mode */
+ EXPECT(PQexitBatchMode(conn), "couldn't exit idle empty batch mode\n");
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "Batch mode not terminated properly\n");
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ EXPECT(PQexitBatchMode(conn), "batch mode exit when not in batch mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "PQexec should succeed after exiting batch mode but failed with: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the batch results */
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from first batch item\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after first result\n");
+
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s instead of sync result, error: %s (line %u)\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* second batch */
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from second batch item\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s from second end batch\n", PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_COMMAND_OK,
+ "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* second query, caused error */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_FATAL_ERROR,
+ "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_ABORTED, "batch should be flagged as aborted but isn't\n");
+
+ /* third query in batch, the second insert */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_ABORTED,
+ "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_ABORTED, "batch should be flagged as aborted but isn't\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_END,
+ "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_ABORTED, "sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* the insert from the second batch */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_COMMAND_OK, "Unexpected result code %s from first item in second batch\n", res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second batch sync */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s from second batch sync\n", res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Expected tuples, got %s: %s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ EXPECT(strcmp(val, "3") == 0, "expected only insert with value 3, got %s", val);
+ }
+
+ EXPECT(PQntuples(res) == 1, "expected 1 result, got %d", PQntuples(res));
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0), "xact start failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ EXPECT(PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0), "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ EXPECT(PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0), "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ EXPECT(PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids), "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ EXPECT(PQsetnonblocking(conn, 1) == 0, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ EXPECT(PQresultStatus(res) == status, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+
+ EXPECT(strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) == 0, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSendQueue(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsetnonblocking(conn, 0) == 0, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "prepare failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "INSERT failed: %s\n", PQerrorMessage(conn));
+
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ EXPECT(PQresultStatus(res) == PGRES_COPY_IN, "COPY: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ EXPECT(formatted < 12 + 1, "Buffer write truncated somehow\n");
+
+ EXPECT(PQputCopyData(conn, buf, formatted) == 1, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+
+ nrows--;
+ }
+
+ EXPECT(PQputCopyEnd(conn, NULL) == 1, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ instr_time start_time, end_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_pipelined(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+
+ printf("batch insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_sequential(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("sequential insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_copy(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("COPY elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQenterBatchMode(conn);
+
+ for (i=0; i < 3; i++) {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchSendQueue(conn);
+
+ i=0;
+ while (true)
+ {
+ int isSingleTuple = 0;
+ ExecStatusType last_est = 0;
+
+ /* Set single row mode for only first 3 SELECT queries */
+ if(i < 3)
+ {
+ r = PQsetSingleRowMode(conn);
+ if (r!=1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+ }
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ EXPECT(isSingleTuple, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ isSingleTuple=0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ last_est = est;
+ }
+ i++;
+ if (last_est == PGRES_BATCH_END) break;
+ }
+ PQexitBatchMode(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if(run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 20da7985c1..b131afaa1a 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -50,7 +50,8 @@ my @contrib_excludes = (
'pgcrypto', 'sepgsql',
'brin', 'test_extensions',
'test_misc', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'snapshot_too_old', 'unsafe_tests',
+ 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
@ 2020-10-01 04:35 ` Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Michael Paquier @ 2020-10-01 04:35 UTC (permalink / raw)
To: Matthieu Garrigues <matthieu.garrigues@gmail.com>; +Cc: Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers
On Mon, Sep 21, 2020 at 07:55:03PM +0200, Matthieu Garrigues wrote:
> I merged PQbatchProcessQueue into PQgetResult.
> One first init call to PQbatchProcessQueue was also required in
> PQsendQueue to have
> PQgetResult ready to read the first batch query.
>
> Tests and documentation are updated accordingly.
The documentation is failing to build, and the patch does not build
correctly on Windows. Could you address that?
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
@ 2020-10-01 06:41 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
0 siblings, 1 reply; 182+ messages in thread
From: Matthieu Garrigues @ 2020-10-01 06:41 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers
On Thu, Oct 1, 2020 at 6:35 AM Michael Paquier <michael@paquier.xyz> wrote:
> The documentation is failing to build, and the patch does not build
> correctly on Windows. Could you address that?
> --
> Michael
Yes I'm on it.
--
Matthieu
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
@ 2020-10-01 08:13 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 182+ messages in thread
From: Matthieu Garrigues @ 2020-10-01 08:13 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers
This patch fixes compilation on windows and compilation of the documentation.
Matthieu Garrigues
On Thu, Oct 1, 2020 at 8:41 AM Matthieu Garrigues
<matthieu.garrigues@gmail.com> wrote:
>
> On Thu, Oct 1, 2020 at 6:35 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> > The documentation is failing to build, and the patch does not build
> > correctly on Windows. Could you address that?
> > --
> > Michael
>
> Yes I'm on it.
>
> --
> Matthieu
Attachments:
libpq-batch-v21.patch (text/x-patch, 87.6K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 92556c7ce0..932561d0c5 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4829,6 +4829,465 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 14.0, but clients using PostgresSQL 14.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending"/>.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the user must call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQgetResult</function> again and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> before retrieving
+ results with <function>PQgetResult</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode"/>.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (exept that PQgetResult
+ return null to indicate that we start returning the results of next query). The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4869,6 +5328,15 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <note>
+ <para>
+ When using batch mode, activate the single row mode on the current batch query by
+ calling <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index cf4653fe0f..a1b46c83ed 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 8afa5a29b4..69bf757e85 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1022,6 +1022,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/include/port/linux.h b/src/include/port/linux.h
index 7a6e46cdbb..6a217aa39f 100644
--- a/src/include/port/linux.h
+++ b/src/include/port/linux.h
@@ -1,22 +1,69 @@
-/* src/include/port/linux.h */
-
-/*
- * As of July 2007, all known versions of the Linux kernel will sometimes
- * return EIDRM for a shmctl() operation when EINVAL is correct (it happens
- * when the low-order 15 bits of the supplied shm ID match the slot number
- * assigned to a newer shmem segment). We deal with this by assuming that
- * EIDRM means EINVAL in PGSharedMemoryIsInUse(). This is reasonably safe
- * since in fact Linux has no excuse for ever returning EIDRM; it doesn't
- * track removed segments in a way that would allow distinguishing them from
- * private ones. But someday that code might get upgraded, and we'd have
- * to have a kernel version test here.
- */
-#define HAVE_LINUX_EIDRM_BUG
-
-/*
- * Set the default wal_sync_method to fdatasync. With recent Linux versions,
- * xlogdefs.h's normal rules will prefer open_datasync, which (a) doesn't
- * perform better and (b) causes outright failures on ext4 data=journal
- * filesystems, because those don't support O_DIRECT.
- */
-#define PLATFORM_DEFAULT_SYNC_METHOD SYNC_METHOD_FDATASYNC
+/* src/include/port/win32.h */
+
+/*
+ * We always rely on the WIN32 macro being set by our build system,
+ * but _WIN32 is the compiler pre-defined macro. So make sure we define
+ * WIN32 whenever _WIN32 is set, to facilitate standalone building.
+ */
+#if defined(_WIN32) && !defined(WIN32)
+#define WIN32
+#endif
+
+/*
+ * Make sure _WIN32_WINNT has the minimum required value.
+ * Leave a higher value in place. When building with at least Visual
+ * Studio 2015 the minimum requirement is Windows Vista (0x0600) to
+ * get support for GetLocaleInfoEx() with locales. For everything else
+ * the minimum version is Windows XP (0x0501).
+ */
+#if defined(_MSC_VER) && _MSC_VER >= 1900
+#define MIN_WINNT 0x0600
+#else
+#define MIN_WINNT 0x0501
+#endif
+
+#if defined(_WIN32_WINNT) && _WIN32_WINNT < MIN_WINNT
+#undef _WIN32_WINNT
+#endif
+
+#ifndef _WIN32_WINNT
+#define _WIN32_WINNT MIN_WINNT
+#endif
+
+/*
+ * We need to prevent <crtdefs.h> from defining a symbol conflicting with
+ * our errcode() function. Since it's likely to get included by standard
+ * system headers, pre-emptively include it now.
+ */
+#if defined(_MSC_VER) || defined(HAVE_CRTDEFS_H)
+#define errcode __msvc_errcode
+#include <crtdefs.h>
+#undef errcode
+#endif
+
+/*
+ * defines for dynamic linking on Win32 platform
+ */
+
+#ifdef BUILDING_DLL
+#define PGDLLIMPORT __declspec (dllexport)
+#else
+#define PGDLLIMPORT __declspec (dllimport)
+#endif
+
+#ifdef _MSC_VER
+#define PGDLLEXPORT __declspec (dllexport)
+#else
+#define PGDLLEXPORT
+#endif
+
+/*
+ * Windows headers don't define this structure, but you can define it yourself
+ * to use the functionality.
+ */
+struct sockaddr_un
+{
+ unsigned short sun_family;
+ char sun_path[108];
+};
+#define HAVE_STRUCT_SOCKADDR_UN 1
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..c8c218b51d 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterBatchMode 180
+PQexitBatchMode 181
+PQbatchSendQueue 182
+PQbatchStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 7bee9dd201..25c79cb7b4 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -535,6 +535,24 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
/*
* pqDropServerData
@@ -554,6 +572,7 @@ pqDropServerData(PGconn *conn)
{
PGnotify *notify;
pgParameterStatus *pstatus;
+ PGcommandQueueEntry *queue;
/* Forget pending notifies */
notify = conn->notifyHead;
@@ -566,6 +585,14 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index eea0237c3a..ff9a6ca253 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -70,7 +72,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1210,7 +1215,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1233,6 +1238,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1331,6 +1343,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1389,31 +1405,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1461,7 +1497,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1479,20 +1588,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1516,6 +1665,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1525,6 +1678,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1637,35 +1807,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1766,14 +1943,308 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ /* error message should be set up already */
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+
+/*
+ * internal function pqBatchProcessQueue
+ *
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+static int
+pqBatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
+/*
+ * PQbatchSendQueue
+ *
+ * End a batch submission.
+ *
+ * It's legal to start submitting another batch immediately, without
+ * waiting for the results of the current batch. There's no need to end batch
+ * mode and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and
+ * including the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED
+ * state. If the whole batch is processed without error, a PGresult with
+ * PGRES_BATCH_END is produced.
+ *
+ * Queries can already have been sent before PQbatchSendQueue is called, but
+ * PQbatchSendQueue need to be called before retrieving command results.
+ *
+ * The connection will remain in batch mode and unavailable for new synchronous
+ * command execution functions until all results from the batch are processed
+ * by the client.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ return 0;
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ return 0;
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqBatchProcessQueue so the user can call start calling getResult.
+ */
+ pqBatchProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In batch mode, once all the result of a query have been
+ * returned, PQgetResult returns NULL to let the user know that
+ * the next batched query is being processed.
+ *
+ * At the end of the batch, returns a end-of-batch result with
+ * PQresultStatus(result) == PGRES_BATCH_END.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1842,9 +2313,38 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, we prepare the processing of the results
+ * of the next query.
+ */
+ pqBatchProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as there
+ * can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -2025,6 +2525,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2219,6 +2726,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2234,6 +2744,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2242,15 +2766,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2260,14 +2787,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2665,6 +3196,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3858,3 +4396,15 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return(pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 9360c541be..2ff3fa4883 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -406,6 +406,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 1696525475..9c37a9c4ff 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -217,10 +217,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -875,6 +883,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 3b6a9fbce3..b6e3cde539 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -15,6 +15,10 @@
#ifndef LIBPQ_FE_H
#define LIBPQ_FE_H
+#ifdef WIN32
+#include <windows.h>
+#endif
+
#ifdef __cplusplus
extern "C"
{
@@ -97,7 +101,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -137,6 +144,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -435,6 +453,12 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1de91ae295..dee4c85382 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -394,6 +416,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -406,6 +429,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -798,6 +831,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -806,6 +844,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index a6d2ffbf9e..287214c544 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..11e8463984
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..d907063f65
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..2c361880fa
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,27 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_batch simple_batch multi_batch batch_abort timings singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'testlibpqbatch', 'dbname=postgres', "$numrows", "$testname" ],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000000..b82634171b
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1003 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "c.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+#define EXPECT(condition, ...) \
+ if (0 == (condition)) \
+ { \
+ fprintf(stderr, __VA_ARGS__); \
+ goto fail; \
+ }
+
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ EXPECT(!PQisnonblocking(conn), "Expected blocking connection mode\n");
+
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0),
+ "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode with work in progress should fail, but succeeded\n");
+
+ EXPECT(PQbatchSendQueue(conn), "Ending a batch failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from first batch item\n", PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when sync result PGRES_BATCH_END expected: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s instead of sync result, error: %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn))
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after end batch call\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ EXPECT(!PQisnonblocking(conn), "Expected blocking connection mode: %u\n", __LINE__);
+
+ EXPECT(PQenterBatchMode(conn), "Unable to enter batch mode\n");
+
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Batch mode not activated properly\n");
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ EXPECT(PQresultStatus(res) == PGRES_FATAL_ERROR, "PQexec should fail in batch mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ EXPECT(PQsendQuery(conn, "SELECT 1") == 0, "PQsendQuery should fail in batch mode but succeeded\n");
+
+ /* Entering batch mode when already in batch mode is OK */
+ EXPECT(PQenterBatchMode(conn), "re-entering batch mode should be a no-op but failed\n");
+
+ EXPECT(!PQisBusy(conn), "PQisBusy should return false when idle in batch, returned true\n");
+
+ /* ok, back to normal command mode */
+ EXPECT(PQexitBatchMode(conn), "couldn't exit idle empty batch mode\n");
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "Batch mode not terminated properly\n");
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ EXPECT(PQexitBatchMode(conn), "batch mode exit when not in batch mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "PQexec should succeed after exiting batch mode but failed with: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the batch results */
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from first batch item\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after first result\n");
+
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s instead of sync result, error: %s (line %u)\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* second batch */
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from second batch item\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s from second end batch\n", PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_COMMAND_OK,
+ "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* second query, caused error */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_FATAL_ERROR,
+ "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_ABORTED, "batch should be flagged as aborted but isn't\n");
+
+ /* third query in batch, the second insert */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_ABORTED,
+ "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_ABORTED, "batch should be flagged as aborted but isn't\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_END,
+ "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_ABORTED, "sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* the insert from the second batch */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_COMMAND_OK, "Unexpected result code %s from first item in second batch\n", res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second batch sync */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s from second batch sync\n", res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Expected tuples, got %s: %s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ EXPECT(strcmp(val, "3") == 0, "expected only insert with value 3, got %s", val);
+ }
+
+ EXPECT(PQntuples(res) == 1, "expected 1 result, got %d", PQntuples(res));
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0), "xact start failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ EXPECT(PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0), "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ EXPECT(PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0), "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ EXPECT(PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids), "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ EXPECT(PQsetnonblocking(conn, 1) == 0, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ EXPECT(PQresultStatus(res) == status, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+
+ EXPECT(strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) == 0, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSendQueue(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsetnonblocking(conn, 0) == 0, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "prepare failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "INSERT failed: %s\n", PQerrorMessage(conn));
+
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ EXPECT(PQresultStatus(res) == PGRES_COPY_IN, "COPY: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ EXPECT(formatted < 12 + 1, "Buffer write truncated somehow\n");
+
+ EXPECT(PQputCopyData(conn, buf, formatted) == 1, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+
+ nrows--;
+ }
+
+ EXPECT(PQputCopyEnd(conn, NULL) == 1, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ instr_time start_time, end_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_pipelined(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+
+ printf("batch insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_sequential(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("sequential insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_copy(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("COPY elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQenterBatchMode(conn);
+
+ for (i=0; i < 3; i++) {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchSendQueue(conn);
+
+ i=0;
+ while (true)
+ {
+ int isSingleTuple = 0;
+ ExecStatusType last_est = 0;
+
+ /* Set single row mode for only first 3 SELECT queries */
+ if(i < 3)
+ {
+ r = PQsetSingleRowMode(conn);
+ if (r!=1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+ }
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ EXPECT(isSingleTuple, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ isSingleTuple=0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ last_est = est;
+ }
+ i++;
+ if (last_est == PGRES_BATCH_END) break;
+ }
+ PQexitBatchMode(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if(run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 20da7985c1..b131afaa1a 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -50,7 +50,8 @@ my @contrib_excludes = (
'pgcrypto', 'sepgsql',
'brin', 'test_extensions',
'test_misc', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'snapshot_too_old', 'unsafe_tests',
+ 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
@ 2020-10-26 19:09 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2020-10-26 19:09 UTC (permalink / raw)
To: Matthieu Garrigues <matthieu.garrigues@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers
I started reading this patch. As a starting point I decided to post an
updated copy (v22), wherein I reverted the overwriting of
src/include/port/linux.h with the win32.h contents (?) and the inclusion
of <windows.h> in libpq-fe.h. If those are needed, some different
solution will have to be found.
Trivial other changes (pgindent etc); nothing of substance. With the
PQtrace() patch I posted at [1] the added test program crashes -- I
don't know if the fault lies in this patch or the other one.
[1] https://postgr.es/m/20201026162313.GA22502@alvherre.pgsql
Attachments:
v22-0001-libpq-batch.patch (text/x-diff, 86.7K) download
From 4d4e86c817599c7370b44c75e3caca9b4d3fea2d Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 26 Oct 2020 13:24:24 -0300
Subject: [PATCH v22] libpq batch
---
doc/src/sgml/libpq.sgml | 468 ++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 4 +
src/interfaces/libpq/fe-connect.c | 27 +
src/interfaces/libpq/fe-exec.c | 630 ++++++++++-
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 16 +-
src/interfaces/libpq/libpq-fe.h | 22 +-
src/interfaces/libpq/libpq-int.h | 46 +-
src/test/modules/Makefile | 1 +
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
.../modules/test_libpq/t/001_libpq_async.pl | 27 +
src/test/modules/test_libpq/testlibpqbatch.c | 1008 +++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 3 +-
src/tools/pgindent/typedefs.list | 3 +
18 files changed, 2253 insertions(+), 46 deletions(-)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index de60281fcb..a768feffd3 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4819,6 +4819,465 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 14.0, but clients using PostgresSQL 14.0 version of libpq can
+ use batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ a connection into batch mode. Enter batch mode with <link
+ linkend="libpq-PQenterBatchMode"><function>PQenterBatchMode(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ Using any synchronous command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error conditions.
+ Functions allowed in batch mode are described in <xref linkend="libpq-batch-sending"/>.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function>. It may eventually exit
+ batch mode with <function>PQexitBatchMode</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchSendQueue"><function>PQbatchSendQueue(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the user must call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null. The result from the next batch entry
+ may then be retrieved using <function>PQgetResult</function> again and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> before retrieving
+ results with <function>PQgetResult</function>. This mode selection is effective
+ only for the query currently being processed. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode"/>.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (exept that PQgetResult
+ return null to indicate that we start returning the results of next query). The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue.
+ Based on available memory, results from socket should be read frequently and
+ there's no need to wait till the batch end to read the results. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQexitBatchMode"><function>PQexitBatchMode(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchSendQueue</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4859,6 +5318,15 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <note>
+ <para>
+ When using batch mode, activate the single row mode on the current batch query by
+ calling <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6329cf0796..49be8b1dbe 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 24f8b3e42e..9ae67387a5 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1026,6 +1026,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..ca86f55652 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterBatchMode 180
+PQexitBatchMode 181
+PQbatchSendQueue 182
+PQbatchStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index b0ca37c2ed..8c95ac7fa5 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -536,6 +536,24 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
/*
* pqDropServerData
@@ -555,6 +573,7 @@ pqDropServerData(PGconn *conn)
{
PGnotify *notify;
pgParameterStatus *pstatus;
+ PGcommandQueueEntry *queue;
/* Forget pending notifies */
notify = conn->notifyHead;
@@ -567,6 +586,14 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index eea0237c3a..a492c13e93 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -70,7 +72,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1210,7 +1215,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1233,6 +1238,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1331,6 +1343,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1389,31 +1405,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1461,7 +1497,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1479,20 +1588,61 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1516,6 +1666,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1525,6 +1679,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1637,35 +1808,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1766,14 +1944,308 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ /* error message should be set up already */
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+
+/*
+ * internal function pqBatchProcessQueue
+ *
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+static int
+pqBatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
+/*
+ * PQbatchSendQueue
+ *
+ * End a batch submission.
+ *
+ * It's legal to start submitting another batch immediately, without
+ * waiting for the results of the current batch. There's no need to end batch
+ * mode and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and
+ * including the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED
+ * state. If the whole batch is processed without error, a PGresult with
+ * PGRES_BATCH_END is produced.
+ *
+ * Queries can already have been sent before PQbatchSendQueue is called, but
+ * PQbatchSendQueue need to be called before retrieving command results.
+ *
+ * The connection will remain in batch mode and unavailable for new synchronous
+ * command execution functions until all results from the batch are processed
+ * by the client.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ return 0;
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ return 0;
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqBatchProcessQueue so the user can call start calling getResult.
+ */
+ pqBatchProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In batch mode, once all the result of a query have been
+ * returned, PQgetResult returns NULL to let the user know that
+ * the next batched query is being processed.
+ *
+ * At the end of the batch, returns a end-of-batch result with
+ * PQresultStatus(result) == PGRES_BATCH_END.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1842,9 +2314,38 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, we prepare the processing of the results of
+ * the next query.
+ */
+ pqBatchProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as
+ * there can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -2025,6 +2526,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2219,6 +2727,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2234,6 +2745,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2242,15 +2767,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2260,14 +2788,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2665,6 +3197,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3858,3 +4397,16 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return (pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 9360c541be..2ff3fa4883 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -406,6 +406,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 1696525475..da38e6aed1 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -217,10 +217,19 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -875,6 +884,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 3b6a9fbce3..efe3437bd4 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -97,7 +97,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -137,6 +140,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -435,6 +449,12 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1de91ae295..d607390118 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -394,6 +416,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -406,6 +429,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -798,6 +831,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -806,6 +844,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index a6d2ffbf9e..287214c544 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..11e8463984
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..d907063f65
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..2c361880fa
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,27 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_batch simple_batch multi_batch batch_abort timings singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'testlibpqbatch', 'dbname=postgres', "$numrows", "$testname" ],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000000..7052303d59
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1008 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "c.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+#define EXPECT(condition, ...) \
+ if (0 == (condition)) \
+ { \
+ fprintf(stderr, __VA_ARGS__); \
+ goto fail; \
+ }
+
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ EXPECT(!PQisnonblocking(conn), "Expected blocking connection mode\n");
+
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0),
+ "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode with work in progress should fail, but succeeded\n");
+
+ EXPECT(PQbatchSendQueue(conn), "Ending a batch failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from first batch item\n", PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when sync result PGRES_BATCH_END expected: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s instead of sync result, error: %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn))
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after end batch call\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ EXPECT(!PQisnonblocking(conn), "Expected blocking connection mode: %u\n", __LINE__);
+
+ EXPECT(PQenterBatchMode(conn), "Unable to enter batch mode\n");
+
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Batch mode not activated properly\n");
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ EXPECT(PQresultStatus(res) == PGRES_FATAL_ERROR, "PQexec should fail in batch mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ EXPECT(PQsendQuery(conn, "SELECT 1") == 0, "PQsendQuery should fail in batch mode but succeeded\n");
+
+ /* Entering batch mode when already in batch mode is OK */
+ EXPECT(PQenterBatchMode(conn), "re-entering batch mode should be a no-op but failed\n");
+
+ EXPECT(!PQisBusy(conn), "PQisBusy should return false when idle in batch, returned true\n");
+
+ /* ok, back to normal command mode */
+ EXPECT(PQexitBatchMode(conn), "couldn't exit idle empty batch mode\n");
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "Batch mode not terminated properly\n");
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ EXPECT(PQexitBatchMode(conn), "batch mode exit when not in batch mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "PQexec should succeed after exiting batch mode but failed with: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the batch results */
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from first batch item\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after first result\n");
+
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s instead of sync result, error: %s (line %u)\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* second batch */
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from second batch item\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s from second end batch\n", PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_COMMAND_OK,
+ "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* second query, caused error */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_FATAL_ERROR,
+ "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_ABORTED, "batch should be flagged as aborted but isn't\n");
+
+ /* third query in batch, the second insert */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_ABORTED,
+ "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_ABORTED, "batch should be flagged as aborted but isn't\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_END,
+ "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_ABORTED, "sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* the insert from the second batch */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_COMMAND_OK, "Unexpected result code %s from first item in second batch\n", res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second batch sync */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s from second batch sync\n", res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Expected tuples, got %s: %s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ EXPECT(strcmp(val, "3") == 0, "expected only insert with value 3, got %s", val);
+ }
+
+ EXPECT(PQntuples(res) == 1, "expected 1 result, got %d", PQntuples(res));
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0), "xact start failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ EXPECT(PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0), "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ EXPECT(PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0), "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ EXPECT(PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids), "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ EXPECT(PQsetnonblocking(conn, 1) == 0, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ EXPECT(PQresultStatus(res) == status, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+
+ EXPECT(strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) == 0, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSendQueue(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsetnonblocking(conn, 0) == 0, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "prepare failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "INSERT failed: %s\n", PQerrorMessage(conn));
+
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ EXPECT(PQresultStatus(res) == PGRES_COPY_IN, "COPY: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ EXPECT(formatted < 12 + 1, "Buffer write truncated somehow\n");
+
+ EXPECT(PQputCopyData(conn, buf, formatted) == 1, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+
+ nrows--;
+ }
+
+ EXPECT(PQputCopyEnd(conn, NULL) == 1, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ instr_time start_time,
+ end_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_pipelined(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+
+ printf("batch insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_sequential(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("sequential insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_copy(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("COPY elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,
+ r;
+
+ /* 1 batch, 3 queries in it */
+ r = PQenterBatchMode(conn);
+
+ for (i = 0; i < 3; i++)
+ {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchSendQueue(conn);
+
+ i = 0;
+ while (true)
+ {
+ int isSingleTuple = 0;
+ ExecStatusType last_est = 0;
+
+ /* Set single row mode for only first 3 SELECT queries */
+ if (i < 3)
+ {
+ r = PQsetSingleRowMode(conn);
+ if (r != 1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+ }
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ EXPECT(isSingleTuple, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ isSingleTuple = 0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ last_est = est;
+ }
+ i++;
+ if (last_est == PGRES_BATCH_END)
+ break;
+ }
+ PQexitBatchMode(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+
+}
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 90594bd41b..634e48ec85 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -50,7 +50,8 @@ my @contrib_excludes = (
'pgcrypto', 'sepgsql',
'brin', 'test_extensions',
'test_misc', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'snapshot_too_old', 'unsafe_tests',
+ 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ff853634bc..0e356efecb 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -215,6 +215,7 @@ BackgroundWorkerHandle
BackgroundWorkerSlot
Barrier
BaseBackupCmd
+BatchInsertStep
BeginDirectModify_function
BeginForeignInsert_function
BeginForeignModify_function
@@ -1543,6 +1544,7 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
@@ -1655,6 +1657,7 @@ PMSignalReason
PMState
POLYGON
PQArgBlock
+PQBatchStatus
PQEnvironmentOption
PQExpBuffer
PQExpBufferData
--
2.20.1
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2020-11-02 15:45 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2020-11-02 15:45 UTC (permalink / raw)
To: Matthieu Garrigues <matthieu.garrigues@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers
In v23 I've gone over docs; discovered that PQgetResults docs were
missing the new values. Added those. No significant other changes yet.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2020-11-02 15:57 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 13:20 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
0 siblings, 2 replies; 182+ messages in thread
From: Alvaro Herrera @ 2020-11-02 15:57 UTC (permalink / raw)
To: Matthieu Garrigues <matthieu.garrigues@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers
On 2020-Nov-02, Alvaro Herrera wrote:
> In v23 I've gone over docs; discovered that PQgetResults docs were
> missing the new values. Added those. No significant other changes yet.
Attachments:
v23-0001-libpq-batch.patch (text/x-diff, 87.5K) download
From 3f305d05000981445fe4dbbb96c2e88ace89f439 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 26 Oct 2020 13:24:24 -0300
Subject: [PATCH v23] libpq batch
---
doc/src/sgml/libpq.sgml | 505 ++++++++
doc/src/sgml/lobj.sgml | 4 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/interfaces/libpq/exports.txt | 4 +
src/interfaces/libpq/fe-connect.c | 27 +
src/interfaces/libpq/fe-exec.c | 630 +++++++++-
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 16 +-
src/interfaces/libpq/libpq-fe.h | 22 +-
src/interfaces/libpq/libpq-int.h | 46 +-
src/test/modules/Makefile | 1 +
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
.../modules/test_libpq/t/001_libpq_async.pl | 27 +
src/test/modules/test_libpq/testlibpqbatch.c | 1013 +++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 3 +-
src/tools/pgindent/typedefs.list | 3 +
18 files changed, 2295 insertions(+), 46 deletions(-)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 9ce32fb39b..839cadfaf0 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3061,6 +3061,30 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-batch-end">
+ <term><literal>PGRES_BATCH_END</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents the end of a batch.
+ This status occurs only when batch mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-batch-aborted">
+ <term><literal>PGRES_BATCH_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a batch that's
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and it will return this status code,
+ until the end of the current batch, at which point it will return
+ <literal>PGRES_BATCH_END</literal> and normal processing can resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4819,6 +4843,478 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch Mode and Query Pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up queries into
+ a pipeline to be executed as a batch on the server. Batching queries
+ allows applications to avoid a client/server round-trip after each
+ query to get the results before issuing the next query.
+ </para>
+
+ <sect2>
+ <title>When to Use Batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks, but
+ pipelining can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid sequence. There is usually less benefit
+ in using batches when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without batching; with batching it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ must introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 14.0, but clients using
+ the PostgreSQL 14 version of <application>libpq</application> can use
+ batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using Batch Mode</title>
+
+ <para>
+ To issue batches the application must switch a connection into batch mode.
+ Enter batch mode with <xref linkend="libpq-PQenterBatchMode"/>
+ or test whether batch mode is active with
+ <xref linkend="libpq-PQbatchStatus"/>.
+ In batch mode only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is not recommended as it
+ may trigger failure in batch processing. Using any synchronous
+ command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error
+ conditions. Functions allowed in batch mode are described in
+ <xref linkend="libpq-batch-sending"/>.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function>. It may
+ eventually exit batch mode with <function>PQexitBatchMode</function>
+ once all results are processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ The client will block trying to send queries to the server, but the server
+ will block trying to send results from queries it has already processed to
+ the client. This only occurs when the client sends enough queries to fill
+ its output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required,
+ even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests using
+ normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a
+ <xref linkend="libpq-PQbatchSendQueue"/>
+ call to mark the end of the batch. The client needs not
+ call <function>PQgetResult</function> immediately after
+ dispatching each operation.
+ <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing Results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To process the result of the first batch entry the application calls
+ <function>PQgetResult</function> and handles each result until
+ <function>PQgetResult</function> returns null.
+ The result from the next batch entry may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_BATCH_END</literal>
+ and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time.
+ <literal>PGRES_BATCH_ABORTED</literal> is emitted during error handling;
+ see <link linkend="libpq-batch-errors">error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries and the expected results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ still to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from
+ socket should be read frequently and there's no need to wait till the
+ batch end to read the results. Batches should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per batch.
+ There's no need to exit batch mode and re-enter it between batches
+ or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending Batch Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and
+ the end batch result has been consumed the application may return
+ to non-batched mode with <xref linkend="libpq-PQexitBatchMode"/>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions Associated with Batch Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application>
+ connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQbatchStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ batch mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in batch mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in aborted
+ batch status. The aborted flag is cleared as soon as the result
+ of the <function>PQbatchSendQueue</function> at the end of the aborted
+ batch is processed. Clients don't usually need this function to
+ verify aborted status, as they can tell that the batch is aborted
+ from the <literal>PGRES_BATCH_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ batch mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <link linkend="libpq-batch-errors"> error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ batch mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ is failed.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4859,6 +5355,15 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <note>
+ <para>
+ When using batch mode, activate the single row mode on the current batch query by
+ calling <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6329cf0796..49be8b1dbe 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 24f8b3e42e..9ae67387a5 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1026,6 +1026,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..ca86f55652 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterBatchMode 180
+PQexitBatchMode 181
+PQbatchSendQueue 182
+PQbatchStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index e7781d010f..91f6b205e4 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -536,6 +536,24 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
/*
* pqDropServerData
@@ -555,6 +573,7 @@ pqDropServerData(PGconn *conn)
{
PGnotify *notify;
pgParameterStatus *pstatus;
+ PGcommandQueueEntry *queue;
/* Forget pending notifies */
notify = conn->notifyHead;
@@ -567,6 +586,14 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index eea0237c3a..cf5d639c2f 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -70,7 +72,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1210,7 +1215,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1233,6 +1238,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1331,6 +1343,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1389,31 +1405,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1461,7 +1497,81 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr,
+ libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1479,20 +1589,61 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1516,6 +1667,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1525,6 +1680,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1637,35 +1809,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1766,14 +1945,307 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ /* error message should be set up already */
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+
+/*
+ * internal function pqBatchProcessQueue
+ *
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+static int
+pqBatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result =
+ PQmakeEmptyPGresult(conn, PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
+/*
+ * PQbatchSendQueue
+ *
+ * End a batch submission.
+ *
+ * It's legal to start submitting another batch immediately, without
+ * waiting for the results of the current batch. There's no need to end batch
+ * mode and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and
+ * including the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED
+ * state. If the whole batch is processed without error, a PGresult with
+ * PGRES_BATCH_END is produced.
+ *
+ * Queries can already have been sent before PQbatchSendQueue is called, but
+ * PQbatchSendQueue need to be called before retrieving command results.
+ *
+ * The connection will remain in batch mode and unavailable for new synchronous
+ * command execution functions until all results from the batch are processed
+ * by the client.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ return 0;
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ return 0;
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqBatchProcessQueue so the user can call start calling getResult.
+ */
+ pqBatchProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In batch mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next batched
+ * query is being processed. At the end of the batch, returns a
+ * end-of-batch result with PQresultStatus(result) == PGRES_BATCH_END.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1842,9 +2314,38 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, we prepare the processing of the results of
+ * the next query.
+ */
+ pqBatchProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as
+ * there can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -2025,6 +2526,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2219,6 +2727,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2234,6 +2745,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2242,15 +2767,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2260,14 +2788,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2665,6 +3197,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3858,3 +4397,16 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return (pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 9360c541be..2ff3fa4883 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -406,6 +406,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 1696525475..da38e6aed1 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -217,10 +217,19 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -875,6 +884,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 3b6a9fbce3..efe3437bd4 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -97,7 +97,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -137,6 +140,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -435,6 +449,12 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1de91ae295..d607390118 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -394,6 +416,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -406,6 +429,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -798,6 +831,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -806,6 +844,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index a6d2ffbf9e..287214c544 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..11e8463984
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..d907063f65
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..2c361880fa
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,27 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_batch simple_batch multi_batch batch_abort timings singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'testlibpqbatch', 'dbname=postgres', "$numrows", "$testname" ],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000000..3fff765ada
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1013 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+
+#include "libpq-fe.h"
+
+#define EXPECT(condition, ...) \
+ if (0 == (condition)) \
+ { \
+ fprintf(stderr, __VA_ARGS__); \
+ goto fail; \
+ }
+
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ EXPECT(!PQisnonblocking(conn), "Expected blocking connection mode\n");
+
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0),
+ "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode with work in progress should fail, but succeeded\n");
+
+ EXPECT(PQbatchSendQueue(conn), "Ending a batch failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from first batch item\n", PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when sync result PGRES_BATCH_END expected: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s instead of sync result, error: %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn))
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after end batch call\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ EXPECT(!PQisnonblocking(conn), "Expected blocking connection mode: %u\n", __LINE__);
+
+ EXPECT(PQenterBatchMode(conn), "Unable to enter batch mode\n");
+
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Batch mode not activated properly\n");
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ EXPECT(PQresultStatus(res) == PGRES_FATAL_ERROR, "PQexec should fail in batch mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ EXPECT(PQsendQuery(conn, "SELECT 1") == 0, "PQsendQuery should fail in batch mode but succeeded\n");
+
+ /* Entering batch mode when already in batch mode is OK */
+ EXPECT(PQenterBatchMode(conn), "re-entering batch mode should be a no-op but failed\n");
+
+ EXPECT(!PQisBusy(conn), "PQisBusy should return false when idle in batch, returned true\n");
+
+ /* ok, back to normal command mode */
+ EXPECT(PQexitBatchMode(conn), "couldn't exit idle empty batch mode\n");
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "Batch mode not terminated properly\n");
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ EXPECT(PQexitBatchMode(conn), "batch mode exit when not in batch mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "PQexec should succeed after exiting batch mode but failed with: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the batch results */
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from first batch item\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after first result\n");
+
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s instead of sync result, error: %s (line %u)\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* second batch */
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from second batch item\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s from second end batch\n", PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_COMMAND_OK,
+ "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* second query, caused error */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_FATAL_ERROR,
+ "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_ABORTED, "batch should be flagged as aborted but isn't\n");
+
+ /* third query in batch, the second insert */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_ABORTED,
+ "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_ABORTED, "batch should be flagged as aborted but isn't\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_END,
+ "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_ABORTED, "sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* the insert from the second batch */
+ EXPECT(((res = PQgetResult(conn)) != NULL) &&
+ PQresultStatus(res) == PGRES_COMMAND_OK,
+ "Unexpected result code %s from first item in second batch\n",
+ res ? PQresStatus(PQresultStatus(res)) : "NULL");
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second batch sync */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s from second batch sync\n", res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Expected tuples, got %s: %s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ EXPECT(strcmp(val, "3") == 0, "expected only insert with value 3, got %s", val);
+ }
+
+ EXPECT(PQntuples(res) == 1, "expected 1 result, got %d", PQntuples(res));
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0), "xact start failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ EXPECT(PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0), "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ EXPECT(PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0), "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ EXPECT(PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids), "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ EXPECT(PQsetnonblocking(conn, 1) == 0, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ EXPECT(PQresultStatus(res) == status, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+
+ EXPECT(strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) == 0, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSendQueue(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsetnonblocking(conn, 0) == 0, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "prepare failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "INSERT failed: %s\n", PQerrorMessage(conn));
+
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ EXPECT(PQresultStatus(res) == PGRES_COPY_IN, "COPY: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ EXPECT(formatted < 12 + 1, "Buffer write truncated somehow\n");
+
+ EXPECT(PQputCopyData(conn, buf, formatted) == 1, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+
+ nrows--;
+ }
+
+ EXPECT(PQputCopyEnd(conn, NULL) == 1, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ instr_time start_time,
+ end_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_pipelined(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+
+ printf("batch insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_sequential(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("sequential insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_copy(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("COPY elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,
+ r;
+
+ /* XXX this one is broken */
+ return;
+
+ /* 1 batch, 3 queries in it */
+ r = PQenterBatchMode(conn);
+
+ for (i = 0; i < 3; i++)
+ {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchSendQueue(conn);
+
+ i = 0;
+ for (;;)
+ {
+ int isSingleTuple = 0;
+ ExecStatusType last_est = 0;
+
+ /* Set single row mode for only first 3 SELECT queries */
+ if (i < 3)
+ {
+ r = PQsetSingleRowMode(conn);
+ if (r != 1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+ }
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ EXPECT(isSingleTuple, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ isSingleTuple = 0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ last_est = est;
+ }
+ i++;
+ if (last_est == PGRES_BATCH_END)
+ break;
+ }
+ PQexitBatchMode(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 90594bd41b..634e48ec85 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -50,7 +50,8 @@ my @contrib_excludes = (
'pgcrypto', 'sepgsql',
'brin', 'test_extensions',
'test_misc', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'snapshot_too_old', 'unsafe_tests',
+ 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 03c4e0fe5b..9b75db962b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -215,6 +215,7 @@ BackgroundWorkerHandle
BackgroundWorkerSlot
Barrier
BaseBackupCmd
+BatchInsertStep
BeginDirectModify_function
BeginForeignInsert_function
BeginForeignModify_function
@@ -1544,6 +1545,7 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
@@ -1656,6 +1658,7 @@ PMSignalReason
PMState
POLYGON
PQArgBlock
+PQBatchStatus
PQEnvironmentOption
PQExpBuffer
PQExpBufferData
--
2.20.1
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2020-11-03 13:20 ` Dave Cramer <davecramer@postgres.rocks>
2020-11-03 13:42 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
1 sibling, 1 reply; 182+ messages in thread
From: Dave Cramer @ 2020-11-03 13:20 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers
Alvaro,
On Mon, 2 Nov 2020 at 10:57, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> On 2020-Nov-02, Alvaro Herrera wrote:
>
> > In v23 I've gone over docs; discovered that PQgetResults docs were
> > missing the new values. Added those. No significant other changes yet.
>
>
>
Thanks for looking at this.
What else does it need to get it in shape to apply?
Dave Cramer
www.postgres.rocks
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 13:20 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
@ 2020-11-03 13:42 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 15:56 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-11-11 22:46 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 182+ messages in thread
From: Alvaro Herrera @ 2020-11-03 13:42 UTC (permalink / raw)
To: Dave Cramer <davecramer@postgres.rocks>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers
Hi Dave,
On 2020-Nov-03, Dave Cramer wrote:
> On Mon, 2 Nov 2020 at 10:57, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> > On 2020-Nov-02, Alvaro Herrera wrote:
> >
> > > In v23 I've gone over docs; discovered that PQgetResults docs were
> > > missing the new values. Added those. No significant other changes yet.
>
> Thanks for looking at this.
>
> What else does it need to get it in shape to apply?
I want to go over the code in depth to grok the design more fully.
It would definitely help if you (and others) could think about the API
being added: Does it fulfill the promises being made? Does it offer the
guarantees that real-world apps want to have? I'm not much of an
application writer myself -- particularly high-traffic apps that would
want to use this. As a driver author I would welcome your insight in
these questions.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 13:20 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-11-03 13:42 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2020-11-03 15:56 ` Dave Cramer <davecramer@postgres.rocks>
2020-11-03 16:44 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
1 sibling, 1 reply; 182+ messages in thread
From: Dave Cramer @ 2020-11-03 15:56 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers
On Tue, 3 Nov 2020 at 08:42, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> Hi Dave,
>
> On 2020-Nov-03, Dave Cramer wrote:
>
> > On Mon, 2 Nov 2020 at 10:57, Alvaro Herrera <alvherre@alvh.no-ip.org>
> wrote:
> >
> > > On 2020-Nov-02, Alvaro Herrera wrote:
> > >
> > > > In v23 I've gone over docs; discovered that PQgetResults docs were
> > > > missing the new values. Added those. No significant other changes
> yet.
> >
> > Thanks for looking at this.
> >
> > What else does it need to get it in shape to apply?
>
> I want to go over the code in depth to grok the design more fully.
>
> It would definitely help if you (and others) could think about the API
> being added: Does it fulfill the promises being made? Does it offer the
> guarantees that real-world apps want to have? I'm not much of an
> application writer myself -- particularly high-traffic apps that would
> want to use this. As a driver author I would welcome your insight in
> these questions.
>
>
I'm sort of in the same boat as you. While I'm closer to the client. I
don't personally write that much client code.
I'd really like to hear from the users here.
Dave Cramer
www.postgres.rocks
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 13:20 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-11-03 13:42 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 15:56 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
@ 2020-11-03 16:44 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
0 siblings, 0 replies; 182+ messages in thread
From: Matthieu Garrigues @ 2020-11-03 16:44 UTC (permalink / raw)
To: Dave Cramer <davecramer@postgres.rocks>; +Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers
I implemented a C++ async HTTP server using this new batch mode and it
provides everything I needed to transparently batch sql requests.
It gives a performance boost between x2 and x3 on this benchmark:
https://www.techempower.com/benchmarks/#section=test&runid=3097dbae-5228-454c-ba2e-2055d3982790&...
I'll ask other users interested in this to review the API.
Matthieu Garrigues
On Tue, Nov 3, 2020 at 4:56 PM Dave Cramer <davecramer@postgres.rocks> wrote:
>
>
>
> On Tue, 3 Nov 2020 at 08:42, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>>
>> Hi Dave,
>>
>> On 2020-Nov-03, Dave Cramer wrote:
>>
>> > On Mon, 2 Nov 2020 at 10:57, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>> >
>> > > On 2020-Nov-02, Alvaro Herrera wrote:
>> > >
>> > > > In v23 I've gone over docs; discovered that PQgetResults docs were
>> > > > missing the new values. Added those. No significant other changes yet.
>> >
>> > Thanks for looking at this.
>> >
>> > What else does it need to get it in shape to apply?
>>
>> I want to go over the code in depth to grok the design more fully.
>>
>> It would definitely help if you (and others) could think about the API
>> being added: Does it fulfill the promises being made? Does it offer the
>> guarantees that real-world apps want to have? I'm not much of an
>> application writer myself -- particularly high-traffic apps that would
>> want to use this. As a driver author I would welcome your insight in
>> these questions.
>>
>
> I'm sort of in the same boat as you. While I'm closer to the client. I don't personally write that much client code.
>
> I'd really like to hear from the users here.
>
>
> Dave Cramer
> www.postgres.rocks
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 13:20 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-11-03 13:42 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2020-11-11 22:46 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 182+ messages in thread
From: Andres Freund @ 2020-11-11 22:46 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Dave Cramer <davecramer@postgres.rocks>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers
Hi,
On 2020-11-03 10:42:34 -0300, Alvaro Herrera wrote:
> It would definitely help if you (and others) could think about the API
> being added: Does it fulfill the promises being made? Does it offer the
> guarantees that real-world apps want to have? I'm not much of an
> application writer myself -- particularly high-traffic apps that would
> want to use this.
Somewhere earlier in this thread there was a patch with support for
batching in pgbench. I think it'd be good to refresh that. Both because
it shows at least some real-world-lite usage of the feature and because
we need a way to stress it to see whether it has unnecessary
bottlenecks.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2020-11-03 17:20 ` David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 09:46 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
1 sibling, 2 replies; 182+ messages in thread
From: David G. Johnston @ 2020-11-03 17:20 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers
On Mon, Nov 2, 2020 at 8:58 AM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:
> On 2020-Nov-02, Alvaro Herrera wrote:
>
> > In v23 I've gone over docs; discovered that PQgetResults docs were
> > missing the new values. Added those. No significant other changes yet.
>
>
Just reading the documentation of this patch, haven't been following the
longer thread:
Given the caveats around blocking mode connections why not just require
non-blocking mode, in a similar fashion to how synchronous functions are
disallowed?
"Batched operations will be executed by the server in the order the client
sends them. The server will send the results in the order the statements
executed."
Maybe:
"The server executes statements, and returns results, in the order the
client sends them."
Using two sentences and relying on the user to mentally link the two "in
the order" descriptions together seems to add unnecessary cognitive load.
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing</link> with sending batch queries, or for small batches may
+ process all results after sending the whole batch.
Suggest: "The client may choose to interleave result processing with
sending batch queries, or wait until the complete batch has been sent."
I would expect to process the results of a batch only after sending the
entire batch to the server. That I don't have to is informative but
knowing when I should avoid doing so, and why, is informative as well. To
the extreme while you can use batch mode and interleave if you just poll
getResult after every command you will make the whole batch thing
pointless. Directing the reader from here to the section "Interleaving
Result Processing and Query Dispatch" seems worth considering. The
dynamics of small sizes and sockets remains a bit unclear as to what will
break (if anything, or is it just process memory on the server) if
interleaving it not performed and sizes are large.
I would suggest placing commentary about "all transactions subsequent to a
failed transaction in a batch are ignored while previous completed
transactions are retained" in the "When to Use Batching". Something like
"Batching is less useful, and more complex, when a single batch contains
multiple transactions (see Error Handling)."
My imagined use case would be to open a batch, start a transaction, send
all of its components, end the transaction, end the batch, check for batch
failure and if it doesn't fail have the option to easily continue without
processing individual pgResults (or if it does fail, have the option to
extract the first error pgResult and continue, ignoring the rest, knowing
that the transaction as a whole was reverted and the batch unapplied).
I've never interfaced with libpq directly. Though given how the existing C
API works what is implemented here seems consistent.
The "queueing up queries into a pipeline to be executed as a batch on the
server" can be read as a client-side behavior where nothing is sent to the
server until the batch has been completed. Reading further it becomes
clear that all it basically is is a sever-side toggle that instructs the
server to continue processing incoming commands even while prior commands
have their results waiting to be ingested by the client.
Batch seems like the user-visible term to describe this feature. Pipeline
seems like an implementation detail that doesn't need to be mentioned in
the documentation - especially given that pipeline doesn't get a mentioned
beyond the first two paragraphs of the chapter and never without being
linked directly to "batch". I would probably leave the indexterm and have
a paragraph describing that batching is implemented using a query pipeline
so that people with the implementation detail on their mind can find this
chapter, but the prose for the user should just stick to batching.
Sorry, that all is a bit unfocused, but the documentation for the user of
the API could be cleaned up a bit and some more words spent on
what trade-offs are being made when using batching versus normal
command-response processing. That said, while I don't see all of this
purely a matter of style I'm also not seeing anything demonstrably wrong
with the documentation at the moment. Hopefully my perspective helps
though, and depending on what happens next I may try and make my thoughts
more concrete with an actual patch.
David J.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
@ 2020-11-12 09:46 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
1 sibling, 0 replies; 182+ messages in thread
From: Matthieu Garrigues @ 2020-11-12 09:46 UTC (permalink / raw)
To: David G. Johnston <david.g.johnston@gmail.com>; +Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers
Hi David,
Thanks for the feedback. I did rework a bit the doc based on your
remarks. Here is the v24 patch.
Matthieu Garrigues
On Tue, Nov 3, 2020 at 6:21 PM David G. Johnston
<david.g.johnston@gmail.com> wrote:
>
> On Mon, Nov 2, 2020 at 8:58 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>>
>> On 2020-Nov-02, Alvaro Herrera wrote:
>>
>> > In v23 I've gone over docs; discovered that PQgetResults docs were
>> > missing the new values. Added those. No significant other changes yet.
>>
>
> Just reading the documentation of this patch, haven't been following the longer thread:
>
> Given the caveats around blocking mode connections why not just require non-blocking mode, in a similar fashion to how synchronous functions are disallowed?
>
> "Batched operations will be executed by the server in the order the client
> sends them. The server will send the results in the order the statements
> executed."
>
> Maybe:
>
> "The server executes statements, and returns results, in the order the client sends them."
>
> Using two sentences and relying on the user to mentally link the two "in the order" descriptions together seems to add unnecessary cognitive load.
>
> + The client <link linkend="libpq-batch-interleave">interleaves result
> + processing</link> with sending batch queries, or for small batches may
> + process all results after sending the whole batch.
>
> Suggest: "The client may choose to interleave result processing with sending batch queries, or wait until the complete batch has been sent."
>
> I would expect to process the results of a batch only after sending the entire batch to the server. That I don't have to is informative but knowing when I should avoid doing so, and why, is informative as well. To the extreme while you can use batch mode and interleave if you just poll getResult after every command you will make the whole batch thing pointless. Directing the reader from here to the section "Interleaving Result Processing and Query Dispatch" seems worth considering. The dynamics of small sizes and sockets remains a bit unclear as to what will break (if anything, or is it just process memory on the server) if interleaving it not performed and sizes are large.
>
> I would suggest placing commentary about "all transactions subsequent to a failed transaction in a batch are ignored while previous completed transactions are retained" in the "When to Use Batching". Something like "Batching is less useful, and more complex, when a single batch contains multiple transactions (see Error Handling)."
>
> My imagined use case would be to open a batch, start a transaction, send all of its components, end the transaction, end the batch, check for batch failure and if it doesn't fail have the option to easily continue without processing individual pgResults (or if it does fail, have the option to extract the first error pgResult and continue, ignoring the rest, knowing that the transaction as a whole was reverted and the batch unapplied). I've never interfaced with libpq directly. Though given how the existing C API works what is implemented here seems consistent.
>
> The "queueing up queries into a pipeline to be executed as a batch on the server" can be read as a client-side behavior where nothing is sent to the server until the batch has been completed. Reading further it becomes clear that all it basically is is a sever-side toggle that instructs the server to continue processing incoming commands even while prior commands have their results waiting to be ingested by the client.
>
> Batch seems like the user-visible term to describe this feature. Pipeline seems like an implementation detail that doesn't need to be mentioned in the documentation - especially given that pipeline doesn't get a mentioned beyond the first two paragraphs of the chapter and never without being linked directly to "batch". I would probably leave the indexterm and have a paragraph describing that batching is implemented using a query pipeline so that people with the implementation detail on their mind can find this chapter, but the prose for the user should just stick to batching.
>
> Sorry, that all is a bit unfocused, but the documentation for the user of the API could be cleaned up a bit and some more words spent on what trade-offs are being made when using batching versus normal command-response processing. That said, while I don't see all of this purely a matter of style I'm also not seeing anything demonstrably wrong with the documentation at the moment. Hopefully my perspective helps though, and depending on what happens next I may try and make my thoughts more concrete with an actual patch.
>
> David J.
>
Attachments:
v24-libpq-batch.patch (text/x-patch, 86.1K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 9ce32fb39b..044e3a705f 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3061,6 +3061,30 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-batch-end">
+ <term><literal>PGRES_BATCH_END</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents the end of a batch.
+ This status occurs only when batch mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-batch-aborted">
+ <term><literal>PGRES_BATCH_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a batch that's
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and it will return this status code,
+ until the end of the current batch, at which point it will return
+ <literal>PGRES_BATCH_END</literal> and normal processing can resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4819,6 +4843,481 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch Mode</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> batch mode allows applications to send a query without
+ having to read the result of the previously sent query. Taking advantage of the batch mode, a client
+ will wait less for the server, since multiple queries/results can be sent/received in a single network
+ transaction.
+
+ While providing a performance boost, writing clients using the batch mode is more complex because it
+ involves managing a queue of pending queries, and finding which result corresponds to which query in the queue.
+ </para>
+
+ <sect2>
+ <title>When to Use Batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks, but
+ pipelining can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid sequence. There is usually less benefit
+ in using batches when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without batching; with batching it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ must introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ Batching is less useful, and more complex, when a single batch contains multiple transactions
+ (see <link linkend="libpq-batch-errors">error handling</link>).
+ </para>
+ </note>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 14.0, but clients using
+ the PostgreSQL 14 version of <application>libpq</application> can use
+ batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using Batch Mode</title>
+
+ <para>
+ To issue batches the application must switch a connection into batch mode.
+ Enter batch mode with <xref linkend="libpq-PQenterBatchMode"/>
+ or test whether batch mode is active with
+ <xref linkend="libpq-PQbatchStatus"/>.
+ In batch mode only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is not recommended as it
+ may trigger failure in batch processing. Using any synchronous
+ command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error
+ conditions. Functions allowed in batch mode are described in
+ <xref linkend="libpq-batch-sending"/>.
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchSendQueue</function>.
+ And to get results, it uses <function>PQgetResult</function>. It may
+ eventually exit batch mode with <function>PQexitBatchMode</function>
+ once all results are processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ The client will block trying to send queries to the server, but the server
+ will block trying to send results from queries it has already processed to
+ the client. This only occurs when the client sends enough queries to fill
+ its output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ Batch mode consumes more memory when send/recv is not done as required,
+ even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests using
+ normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a
+ <xref linkend="libpq-PQbatchSendQueue"/>
+ call to mark the end of the batch. The client needs not
+ call <function>PQgetResult</function> immediately after
+ dispatching each operation.
+ <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the client sends them.
+ The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of the first batch entry the application calls
+ <function>PQgetResult</function> and handles each result until
+ <function>PQgetResult</function> returns null.
+ The result from the next batch entry may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_BATCH_END</literal>
+ and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time.
+ <literal>PGRES_BATCH_ABORTED</literal> is emitted during error handling;
+ see <link linkend="libpq-batch-errors">error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries and the expected results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ still to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from
+ socket should be read frequently and there's no need to wait till the
+ batch end to read the results. Batches should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per batch.
+ There's no need to exit batch mode and re-enter it between batches
+ or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending Batch Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and
+ the end batch result has been consumed the application may return
+ to non-batched mode with <xref linkend="libpq-PQexitBatchMode"/>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions Associated with Batch Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application>
+ connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQbatchStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ batch mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in batch mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in aborted
+ batch status. The aborted flag is cleared as soon as the result
+ of the <function>PQbatchSendQueue</function> at the end of the aborted
+ batch is processed. Clients don't usually need this function to
+ verify aborted status, as they can tell that the batch is aborted
+ from the <literal>PGRES_BATCH_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ batch mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <link linkend="libpq-batch-errors"> error handling</link>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ batch mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ is failed.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4859,6 +5358,15 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <note>
+ <para>
+ When using batch mode, activate the single row mode on the current batch query by
+ calling <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6329cf0796..49be8b1dbe 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 24f8b3e42e..9ae67387a5 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1026,6 +1026,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..ca86f55652 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterBatchMode 180
+PQexitBatchMode 181
+PQbatchSendQueue 182
+PQbatchStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index e7781d010f..91f6b205e4 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -536,6 +536,24 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
/*
* pqDropServerData
@@ -555,6 +573,7 @@ pqDropServerData(PGconn *conn)
{
PGnotify *notify;
pgParameterStatus *pstatus;
+ PGcommandQueueEntry *queue;
/* Forget pending notifies */
notify = conn->notifyHead;
@@ -567,6 +586,14 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index eea0237c3a..cf5d639c2f 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -70,7 +72,10 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
-
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
* Space management for PGresult.
@@ -1210,7 +1215,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1233,6 +1238,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1331,6 +1343,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1389,31 +1405,51 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1461,7 +1497,81 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr,
+ libpq_gettext("tried to recycle non-dangling command queue entry"));
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1479,20 +1589,61 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1516,6 +1667,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1525,6 +1680,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1637,35 +1809,42 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1766,14 +1945,307 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return 0;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns 1 if batch mode ended.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ goto exitFailed;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ goto exitFailed;
+ default:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ goto exitFailed;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ goto sendFailed;
+ return 1;
+
+sendFailed:
+ /* error message should be set up already */
+
+exitFailed:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, Failed to exit batch mode"));
+ return 0;
+}
+
+
+/*
+ * internal function pqBatchProcessQueue
+ *
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+static int
+pqBatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result =
+ PQmakeEmptyPGresult(conn, PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
+/*
+ * PQbatchSendQueue
+ *
+ * End a batch submission.
+ *
+ * It's legal to start submitting another batch immediately, without
+ * waiting for the results of the current batch. There's no need to end batch
+ * mode and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and
+ * including the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED
+ * state. If the whole batch is processed without error, a PGresult with
+ * PGRES_BATCH_END is produced.
+ *
+ * Queries can already have been sent before PQbatchSendQueue is called, but
+ * PQbatchSendQueue need to be called before retrieving command results.
+ *
+ * The connection will remain in batch mode and unavailable for new synchronous
+ * command execution functions until all results from the batch are processed
+ * by the client.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ return 0;
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ return 0;
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqBatchProcessQueue so the user can call start calling getResult.
+ */
+ pqBatchProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In batch mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next batched
+ * query is being processed. At the end of the batch, returns a
+ * end-of-batch result with PQresultStatus(result) == PGRES_BATCH_END.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1842,9 +2314,38 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, we prepare the processing of the results of
+ * the next query.
+ */
+ pqBatchProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as
+ * there can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -2025,6 +2526,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2219,6 +2727,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2234,6 +2745,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2242,15 +2767,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2260,14 +2788,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
* Give the data a push. In nonblock mode, don't complain if we're unable
* to send it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2665,6 +3197,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -3858,3 +4397,16 @@ PQunescapeBytea(const unsigned char *strtext, size_t *retbuflen)
*retbuflen = buflen;
return tmpbuf;
}
+
+/* pqBatchFlush
+ * In batch mode, data will be flushed only when the out buffer reaches the threshold value.
+ * In non-batch mode, data will be flushed all the time.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return (pqFlush(conn));
+ return 0; /* Just to keep compiler quiet */
+}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 9360c541be..2ff3fa4883 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -406,6 +406,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 1696525475..da38e6aed1 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -217,10 +217,19 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -875,6 +884,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 3b6a9fbce3..efe3437bd4 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -97,7 +97,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -137,6 +140,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -435,6 +449,12 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1de91ae295..d607390118 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -394,6 +416,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -406,6 +429,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -798,6 +831,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
@@ -806,6 +844,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index a6d2ffbf9e..287214c544 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..11e8463984
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..d907063f65
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..2c361880fa
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,27 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_batch simple_batch multi_batch batch_abort timings singlerowmode);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'testlibpqbatch', 'dbname=postgres', "$numrows", "$testname" ],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000000..3fff765ada
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1013 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+
+#include "libpq-fe.h"
+
+#define EXPECT(condition, ...) \
+ if (0 == (condition)) \
+ { \
+ fprintf(stderr, __VA_ARGS__); \
+ goto fail; \
+ }
+
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_singlerowmode(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ EXPECT(!PQisnonblocking(conn), "Expected blocking connection mode\n");
+
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0),
+ "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode with work in progress should fail, but succeeded\n");
+
+ EXPECT(PQbatchSendQueue(conn), "Ending a batch failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from first batch item\n", PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when sync result PGRES_BATCH_END expected: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s instead of sync result, error: %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn))
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after end batch call\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ EXPECT(!PQisnonblocking(conn), "Expected blocking connection mode: %u\n", __LINE__);
+
+ EXPECT(PQenterBatchMode(conn), "Unable to enter batch mode\n");
+
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Batch mode not activated properly\n");
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ EXPECT(PQresultStatus(res) == PGRES_FATAL_ERROR, "PQexec should fail in batch mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ EXPECT(PQsendQuery(conn, "SELECT 1") == 0, "PQsendQuery should fail in batch mode but succeeded\n");
+
+ /* Entering batch mode when already in batch mode is OK */
+ EXPECT(PQenterBatchMode(conn), "re-entering batch mode should be a no-op but failed\n");
+
+ EXPECT(!PQisBusy(conn), "PQisBusy should return false when idle in batch, returned true\n");
+
+ /* ok, back to normal command mode */
+ EXPECT(PQexitBatchMode(conn), "couldn't exit idle empty batch mode\n");
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "Batch mode not terminated properly\n");
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ EXPECT(PQexitBatchMode(conn), "batch mode exit when not in batch mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "PQexec should succeed after exiting batch mode but failed with: %s\n", PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the batch results */
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from first batch item\n", PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "PQgetResult returned something extra after first result\n");
+
+ EXPECT(!PQexitBatchMode(conn), "exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s instead of sync result, error: %s (line %u)\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* second batch */
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Unexpected result code %s from second batch item\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ EXPECT(res != NULL, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s from second end batch\n", PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ EXPECT(PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0), "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchSendQueue(conn), "Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_COMMAND_OK,
+ "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* second query, caused error */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_FATAL_ERROR,
+ "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_ABORTED, "batch should be flagged as aborted but isn't\n");
+
+ /* third query in batch, the second insert */
+
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_ABORTED,
+ "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_ABORTED, "batch should be flagged as aborted but isn't\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_END,
+ "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_ABORTED, "sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* the insert from the second batch */
+ EXPECT(((res = PQgetResult(conn)) != NULL) &&
+ PQresultStatus(res) == PGRES_COMMAND_OK,
+ "Unexpected result code %s from first item in second batch\n",
+ res ? PQresStatus(PQresultStatus(res)) : "NULL");
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second batch sync */
+ EXPECT(((res = PQgetResult(conn)) != NULL) && PQresultStatus(res) == PGRES_BATCH_END, "Unexpected result code %s from second batch sync\n", res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ EXPECT(PQgetResult(conn) == NULL, "Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ EXPECT(PQbatchStatus(conn) != PQBATCH_MODE_OFF, "Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQbatchStatus(conn) == PQBATCH_MODE_OFF, "exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ EXPECT(PQresultStatus(res) == PGRES_TUPLES_OK, "Expected tuples, got %s: %s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ EXPECT(strcmp(val, "3") == 0, "expected only insert with value 3, got %s", val);
+ }
+
+ EXPECT(PQntuples(res) == 1, "expected 1 result, got %d", PQntuples(res));
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ EXPECT(PQenterBatchMode(conn), "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0), "xact start failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ EXPECT(PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0), "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ EXPECT(PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0), "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ EXPECT(PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids), "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ EXPECT(PQsetnonblocking(conn, 1) == 0, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ EXPECT(PQresultStatus(res) == status, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+
+ EXPECT(strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) == 0, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSendQueue(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ EXPECT(PQexitBatchMode(conn), "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+
+ EXPECT(PQsetnonblocking(conn, 0) == 0, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "prepare failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "INSERT failed: %s\n", PQerrorMessage(conn));
+
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ EXPECT(PQresultStatus(res) == PGRES_COPY_IN, "COPY: %s\n", PQerrorMessage(conn));
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ EXPECT(formatted < 12 + 1, "Buffer write truncated somehow\n");
+
+ EXPECT(PQputCopyData(conn, buf, formatted) == 1, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+
+ nrows--;
+ }
+
+ EXPECT(PQputCopyEnd(conn, NULL) == 1, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ EXPECT(PQresultStatus(res) == PGRES_COMMAND_OK, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ instr_time start_time,
+ end_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_pipelined(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+
+ printf("batch insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_sequential(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("sequential insert elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ INSTR_TIME_SET_CURRENT(start_time);
+ batch_insert_copy(conn, number_of_rows);
+ INSTR_TIME_SET_CURRENT(end_time);
+ INSTR_TIME_SUBTRACT(end_time, start_time);
+ printf("COPY elapsed: %.8f ms\n",
+ INSTR_TIME_GET_MILLISEC(end_time));
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings|singlerowmode\n");
+ exit(1);
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i,
+ r;
+
+ /* XXX this one is broken */
+ return;
+
+ /* 1 batch, 3 queries in it */
+ r = PQenterBatchMode(conn);
+
+ for (i = 0; i < 3; i++)
+ {
+ r = PQsendQueryParams(conn,
+ "SELECT 1",
+ 0,
+ NULL,
+ NULL,
+ NULL,
+ NULL,
+ 0);
+ }
+ PQbatchSendQueue(conn);
+
+ i = 0;
+ for (;;)
+ {
+ int isSingleTuple = 0;
+ ExecStatusType last_est = 0;
+
+ /* Set single row mode for only first 3 SELECT queries */
+ if (i < 3)
+ {
+ r = PQsetSingleRowMode(conn);
+ if (r != 1)
+ {
+ fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+ }
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ fprintf(stderr, "Result status: %d (%s) for i=%d", est, PQresStatus(est), i);
+ if (est == PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ EXPECT(isSingleTuple, " Expected to follow PGREG_SINGLE_TUPLE, but received PGRES_TUPLES_OK directly instead\n");
+ isSingleTuple = 0;
+ }
+ else if (est == PGRES_SINGLE_TUPLE)
+ {
+ isSingleTuple = 1;
+ fprintf(stderr, ", single tuple: %d\n", PQntuples(res));
+ }
+ else if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, ", end of batch reached\n");
+ }
+ else if (est != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, ", error: %s\n", PQresultErrorMessage(res));
+ }
+ PQclear(res);
+ last_est = est;
+ }
+ i++;
+ if (last_est == PGRES_BATCH_END)
+ break;
+ }
+ PQexitBatchMode(conn);
+ PQclear(res);
+ res = NULL;
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_singlerowmode = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_singlerowmode = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "singlerowmode") == 0)
+ run_singlerowmode = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_singlerowmode)
+ test_singlerowmode(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 90594bd41b..634e48ec85 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -50,7 +50,8 @@ my @contrib_excludes = (
'pgcrypto', 'sepgsql',
'brin', 'test_extensions',
'test_misc', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'snapshot_too_old', 'unsafe_tests',
+ 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 0e4bb4f07f..4c7709c2e6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -215,6 +215,7 @@ BackgroundWorkerHandle
BackgroundWorkerSlot
Barrier
BaseBackupCmd
+BatchInsertStep
BeginDirectModify_function
BeginForeignInsert_function
BeginForeignModify_function
@@ -1544,6 +1545,7 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
@@ -1656,6 +1658,7 @@ PMSignalReason
PMState
POLYGON
PQArgBlock
+PQBatchStatus
PQEnvironmentOption
PQExpBuffer
PQExpBufferData
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
@ 2020-11-12 13:40 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
1 sibling, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2020-11-12 13:40 UTC (permalink / raw)
To: David G. Johnston <david.g.johnston@gmail.com>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>
(Adding previous reviewers to CC)
On 2020-Nov-03, David G. Johnston wrote:
> Given the caveats around blocking mode connections why not just require
> non-blocking mode, in a similar fashion to how synchronous functions are
> disallowed?
This is a very good question. Why indeed? Does anybody have a good
answer to this? If not, I propose we just require that non-blocking
mode is in use in order for batch mode to be used.
I've been doing a review pass over this patch and have an updated
version, which I intend to share later today (after I fix what appears
to be a misunderstanding in the "singlerow" test in testlibpqbatch.c)
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2020-11-14 00:37 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 14:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
0 siblings, 2 replies; 182+ messages in thread
From: Alvaro Herrera @ 2020-11-14 00:37 UTC (permalink / raw)
To: David G. Johnston <david.g.johnston@gmail.com>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; Heikki Linnakangas <heikki.linnakangas@iki.fi>
Here's a v25.
I made a few more changes to the docs per David's suggestions; I also
reordered the sections quite a bit. It's now like this:
* Batch Mode
* Using Batch Mode
* Issuing Queries
* Processing Results
* Error Handling
* Interleaving Result Processing and Query Dispatch
* Ending Batch Mode
* Functions Associated with Batch Mode
* When to Use Batching
To me as a reader, this makes more sense, but if you disagree, I think
we should discuss further changes. (For example, maybe we should move
the "Functions" section at the end?) The original had "When to Use
Batching" at the start, but it seemed to me that the points it's making
are not as critical as understanding what it is.
I reworked the test program to better fit the TAP model; I found that if
one test mecha failed in whatever way, the connection would be in a
weird state and cause the next test to also fail. I changed so that it
runs one test and exits; then the t/001_libpq_async.pl file (hmm, need
to rename to 001_batch.pl I guess) calls it once for each test.
I adapted the test code to our code style. I also removed the "timings"
stuff; I think that's something better left to pgbench.
(I haven't looked at Daniel's pgbench stuff yet, but I will do that
next.)
While looking at how the tests worked, I gave a hard stare at the new
libpq code and cleaned it up also. There's a lot of minor changes, but
nothing truly substantial. I moved the code around a lot, to keep
things where grouped together they belong.
I'm not 100% clear on things like PGconn->batch_status and how
PGconn->asyncStatus works. Currently everything seems to work
correctly, but I'm worried that because we add new status values to
asyncStatus, some existing code might not be doing everything correctly
(for example when changing to/from ASYNC_BUSY in some cases, are we 100%
we shouldn't be changing to ASYNC_QUEUED?)
While looking this over I noticed a thread from 2014[1] where Matt
Newell tried to implement this stuff and apparently the main review
comment he got before abandoning the patch was that the user would like
a way to access the query that corresponded to each result. The current
patch set does not address that need; the approach to this problem is
that it's on the application's head to keep track of this. Honestly I
don't understand why it would be otherwise ... I'm not sure that it
makes sense to expect that the application is stupid enough that it
doesn't keep track in which order it sent things, but bright enough to
keep pointers to the queries it sent (??). So this seems okay to me.
But added Heikki and Claudio to CC because of that old thread.
[1] https://postgr.es/m/2059418.jZQvL3lH90@obsidian
Attachments:
v25-libpq-batch.patch (text/x-diff, 85.7K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 73c6a6ea4d..60477ca85c 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3061,6 +3061,30 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-batch-end">
+ <term><literal>PGRES_BATCH_END</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents the end of a batch.
+ This status occurs only when batch mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-batch-aborted">
+ <term><literal>PGRES_BATCH_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a batch that's
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and it will return this status code,
+ until the end of the current batch, at which point it will return
+ <literal>PGRES_BATCH_END</literal> and normal processing can resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4819,6 +4843,482 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch Mode</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> batch mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the batch mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While batch mode provides a significant performance boost, writing
+ clients using the batch mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using Batch Mode</title>
+
+ <para>
+ To issue batches the application must switch a connection into batch mode.
+ Enter batch mode with <xref linkend="libpq-PQenterBatchMode"/>
+ or test whether batch mode is active with
+ <xref linkend="libpq-PQbatchStatus"/>.
+ In batch mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is not recommended as it
+ may trigger failure in batch processing. Using any synchronous
+ command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error
+ conditions.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill its output buffer and the server's receive
+ buffer before switching to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ <para>
+ Batch mode consumes more memory when send/receive is not done as required,
+ even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests using
+ normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a
+ <xref linkend="libpq-PQbatchSendQueue"/>
+ call to mark the end of the batch. The client needs not
+ call <function>PQgetResult</function> immediately after
+ dispatching each operation.
+ <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server may begin executing the batch before all
+ commands in the batch are queued and the end of batch command is sent.
+ If any statement encounters an error the server aborts the current
+ transaction and skips processing the rest of the batch.
+ Query processing resumes after the end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one batch query, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next batch query may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the batch have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_BATCH_END</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ batch has been sent, or interleave that with sending further batch
+ queries; see <xref linkend="libpq-batch-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_BATCH_END</literal>
+ and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time.
+ <literal>PGRES_BATCH_ABORTED</literal> is emitted during error handling;
+ see <xref linkend="libpq-batch-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries and the expected results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ still to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from
+ socket should be read frequently and there's no need to wait till the
+ batch end to read the results. Batches should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per batch.
+ There's no need to exit batch mode and re-enter it between batches
+ or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending Batch Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and
+ the end batch result has been consumed the application may return
+ to non-batched mode with <xref linkend="libpq-PQexitBatchMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-batch-functions">
+ <title>Functions Associated with Batch Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application>
+ connection.
+<synopsis>
+int PQbatchStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQbatchStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ batch mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in batch mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in aborted
+ batch status. The aborted flag is cleared as soon as the result
+ of the <function>PQbatchSendQueue</function> at the end of the aborted
+ batch is processed. Clients don't usually need this function to
+ verify aborted status, as they can tell that the batch is aborted
+ from the <literal>PGRES_BATCH_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ batch mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-batch-errors"/>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ batch mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ is failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-batch-tips">
+ <title>When to Use Batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks, but
+ pipelining can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid sequence. There is usually less benefit
+ in using batches when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without batching; with batching it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ must introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Batching is less useful, and more complex, when a single batch contains
+ multiple transactions (see <xref linkend="libpq-batch-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 14.0, but clients using
+ the PostgreSQL 14 version of <application>libpq</application> can use
+ batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4859,6 +5359,17 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <!-- XXX Isn't this note merely repeating what was said above?
+ I think it should be removed. -->
+ <note>
+ <para>
+ When using batch mode, activate the single-row mode on the current
+ batch query by calling <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6329cf0796..49be8b1dbe 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 24f8b3e42e..9ae67387a5 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1026,6 +1026,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..ca86f55652 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterBatchMode 180
+PQexitBatchMode 181
+PQbatchSendQueue 182
+PQbatchStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index e7781d010f..c673bc405f 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -536,6 +536,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -555,6 +572,7 @@ pqDropServerData(PGconn *conn)
{
PGnotify *notify;
pgParameterStatus *pstatus;
+ PGcommandQueueEntry *queue;
/* Forget pending notifies */
notify = conn->notifyHead;
@@ -567,6 +585,14 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ pqFreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ pqFreeCommandQueue(queue);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -6699,6 +6725,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+int
+PQbatchStatus(const PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index eea0237c3a..93971857d5 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -70,6 +72,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static int pqBatchProcessQueue(PGconn *conn);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
@@ -1210,7 +1217,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1233,6 +1240,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return 0;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1263,10 +1277,11 @@ PQsendQuery(PGconn *conn, const char *query)
conn->last_query = strdup(query);
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in batch mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
{
/* error message should be set up already */
return 0;
@@ -1331,6 +1346,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1362,6 +1381,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc batch memory before doing anything */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', false, conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1389,31 +1417,47 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in batch mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1461,7 +1505,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1479,20 +1524,62 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1516,6 +1603,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1525,6 +1616,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1637,35 +1745,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in batch mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1766,14 +1882,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In batch mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next batched
+ * query is being processed. At the end of the batch, returns a
+ * end-of-batch result with PQresultStatus(result) == PGRES_BATCH_END.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1842,9 +1961,38 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, we prepare the processing of the results of
+ * the next query.
+ */
+ pqBatchProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as
+ * there can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -2025,6 +2173,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2219,6 +2374,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2234,6 +2392,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2242,32 +2412,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in batch mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2665,6 +2843,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2685,6 +2870,377 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Batch mode support ======== */
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of
+ * batched commands may be sent while in batch mode. Batch mode can be exited
+ * by calling PQexitBatchMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in batch mode */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot enter batch mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Returns 1 in success (batch mode was ended, or not in batch mode).
+ *
+ * Returns 0 if in batch mode and cannot be ended yet.
+ * Error message will be set.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot exit batch mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot exit batch mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("command queue not clean"));
+ return 0;
+ }
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * internal function pqBatchProcessQueue
+ *
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+static int
+pqBatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ printfPQExpBuffer(&conn->errorMessage,
+ "internal error, COPY in batch mode");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ /* should be unreachable */
+ printfPQExpBuffer(&conn->errorMessage,
+ "internal error, IDLE in batch mode");
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQexitBatchMode() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode XXX why?? */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result =
+ PQmakeEmptyPGresult(conn, PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
+/*
+ * PQbatchSendQueue
+ * End a batch submission.
+ *
+ * It's legal to start submitting another batch immediately, without
+ * waiting for the results of the current batch. There's no need to end batch
+ * mode and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and
+ * including the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED
+ * state. If the whole batch is processed without error, a PGresult with
+ * PGRES_BATCH_END is produced.
+ *
+ * Queries can already have been sent before PQbatchSendQueue is called, but
+ * PQbatchSendQueue need to be called before retrieving command results.
+ *
+ * The connection will remain in batch mode and unavailable for new synchronous
+ * command execution functions until all results from the batch are processed
+ * by the client.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot send batch when not in batch mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ /* should be unreachable */
+ printfPQExpBuffer(&conn->errorMessage,
+ "internal error: cannot send batch when idle\n");
+ return 0;
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ printfPQExpBuffer(&conn->errorMessage,
+ "internal error: cannot send batch while in COPY\n");
+ return 0;
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqBatchProcessQueue so the user can call start calling getResult.
+ */
+ pqBatchProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3285,6 +3841,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqBatchFlush
+ *
+ * In batch mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-batch mode, data will be flushed all the time.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return (pqFlush(conn));
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 9360c541be..2ff3fa4883 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -406,6 +406,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 1696525475..da38e6aed1 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -217,10 +217,19 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -875,6 +884,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 3b6a9fbce3..fcdd887958 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -97,7 +97,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -137,6 +140,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -328,6 +342,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern int PQbatchStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -435,6 +450,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1de91ae295..d2b26f2299 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -394,6 +416,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -406,6 +429,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -798,6 +831,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index a6d2ffbf9e..287214c544 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..11e8463984
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..6d3a0ea4d5
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/test_libpq/Makefile
+
+PROGRAM = testlibpqbatch
+OBJS = testlibpqbatch.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..0b27896a2a
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_batch simple_batch multi_batch batch_abort
+ batch_insert singlerow);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'testlibpqbatch', $testname, $node->connstr('postgres'), $numrows ],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000000..7157e6cb90
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,963 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ * Verify libpq batch execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "testlibpqbatch";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("Unable to enter batch mode\n");
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Batch mode not activated properly\n");
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in batch mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in batch mode but succeeded\n");
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("re-entering batch mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in batch, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty batch mode\n");
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ pg_fatal("Batch mode not terminated properly\n");
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("batch mode exit when not in batch mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitBatchMode(conn) != 0)
+ pg_fatal("exiting batch mode with work in progress should fail, but succeeded\n");
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Ending a batch failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a batch item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQexitBatchMode(conn) != 0)
+ pg_fatal("exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_BATCH_END expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after end batch call\n");
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("attempt to exit batch mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ pg_fatal("Exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the batch results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a batch item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitBatchMode(conn) != 0)
+ pg_fatal("exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* second batch */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a batch item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a batch item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("attempt to exit batch mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ pg_fatal("exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Sending first batch failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-batch insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a batch aborted message for the second insert, a batch-end, then a
+ * command-ok and a batch-ok for the second batch operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ pg_fatal("batch should be flagged as aborted but isn't\n");
+
+ /* third query in batch, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_BATCH_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ pg_fatal("batch should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in batch */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /*
+ * The end of a failed batch is a PGRES_BATCH_END.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code from first batch sync\n"
+ "Expected PGRES_BATCH_END, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* XXX why do we have a NULL result after PGRES_BATCH_END? */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /* the insert from the second batch */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second batch\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second batch sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code %s from second batch sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("attempt to exit batch mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ pg_fatal("exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd batch
+ * - insert applied from 2nd batch
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_batch_insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+test_batch_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSendQueue(conn) == 1)
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("attempt to exit batch mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool batch_ended = false;
+
+ /* 1 batch, 3 queries in it */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQbatchSendQueue(conn);
+
+ for (i = 0; !batch_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, "end of batch reached\n");
+ batch_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!batch_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("failed to end batch mode: %s\n", PQerrorMessage(conn));
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's batch-mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_batch|simple_batch|multi_batch|batch_abort|\n");
+ fprintf(stderr, " singlerow|batch_insert\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (strcmp(argv[1], "disallowed_in_batch") == 0)
+ test_disallowed_in_batch(conn);
+ else if (strcmp(argv[1], "simple_batch") == 0)
+ test_simple_batch(conn);
+ else if (strcmp(argv[1], "multi_batch") == 0)
+ test_multi_batch(conn);
+ else if (strcmp(argv[1], "batch_abort") == 0)
+ test_batch_abort(conn);
+ else if (strcmp(argv[1], "batch_insert") == 0)
+ test_batch_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 90594bd41b..634e48ec85 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -50,7 +50,8 @@ my @contrib_excludes = (
'pgcrypto', 'sepgsql',
'brin', 'test_extensions',
'test_misc', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'snapshot_too_old', 'unsafe_tests',
+ 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 0e4bb4f07f..4c7709c2e6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -215,6 +215,7 @@ BackgroundWorkerHandle
BackgroundWorkerSlot
Barrier
BaseBackupCmd
+BatchInsertStep
BeginDirectModify_function
BeginForeignInsert_function
BeginForeignModify_function
@@ -1544,6 +1545,7 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
@@ -1656,6 +1658,7 @@ PMSignalReason
PMState
POLYGON
PQArgBlock
+PQBatchStatus
PQEnvironmentOption
PQExpBuffer
PQExpBufferData
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2020-11-14 14:42 ` Daniel Verite <daniel@manitou-mail.org>
2020-11-18 17:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
1 sibling, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2020-11-14 14:42 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: David G. Johnston <david.g.johnston@gmail.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael.paquier@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; Heikki Linnakangas <heikki.linnakangas@iki.fi>
Alvaro Herrera wrote:
> I adapted the test code to our code style. I also removed the "timings"
> stuff; I think that's something better left to pgbench.
>
> (I haven't looked at Daniel's pgbench stuff yet, but I will do that
> next.)
The patch I posted in [1] was pretty simple, but at the time, query
results were always discarded. Now that pgbench can instantiate
variables from query results, a script can do:
select 1 as var \gset
select :var;
This kind of sequence wouldn't work in batch mode since it
sends queries before getting results of previous queries.
So maybe \gset should be rejected when inside a batch section.
Or alternatively pgbench should collect results before a variable is
reinjected into a query, thereby stalling the pipeline.
To do this only when necessary, it would have to track read-write
dependencies among variables, which seems overly complicated though.
[1]
https://www.postgresql.org/message-id/b4e34135-2bd9-4b8a-94ca-27d760da26d7@manitou-mail.org
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: https://www.manitou-mail.org
Twitter: @DanielVerite
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 14:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2020-11-18 17:51 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-18 23:17 ` Re: PATCH: Batch/pipelining support for libpq Andres Freund <andres@anarazel.de>
2020-11-23 21:58 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
0 siblings, 2 replies; 182+ messages in thread
From: Alvaro Herrera @ 2020-11-18 17:51 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: David G. Johnston <david.g.johnston@gmail.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael.paquier@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; Heikki Linnakangas <heikki.linnakangas@iki.fi>
On 2020-Nov-14, Daniel Verite wrote:
> The patch I posted in [1] was pretty simple, but at the time, query
> results were always discarded. Now that pgbench can instantiate
> variables from query results, a script can do:
> select 1 as var \gset
> select :var;
> This kind of sequence wouldn't work in batch mode since it
> sends queries before getting results of previous queries.
>
> So maybe \gset should be rejected when inside a batch section.
Hah.
Hacking pgbench extensively is beyond what I'm willing to do for this
feature at this time. Making \gset rejected in a batch section sounds
simple enough and supports \beginbatch et al sufficiently to compare
performance, so I'm OK with a patch that does that. That'd be a small
extension to your previous patch, if I understand correctly.
If you or others want to send patches to extend batch support with
read-write tracking for variables, feel free, but I hereby declare that
I'm not taking immediate responsibility for getting them committed.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 14:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2020-11-18 17:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2020-11-18 23:17 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 182+ messages in thread
From: Andres Freund @ 2020-11-18 23:17 UTC (permalink / raw)
To: pgsql-hackers@lists.postgresql.org
Hi
On Wed, Nov 18, 2020, at 09:51, Alvaro Herrera wrote:
> On 2020-Nov-14, Daniel Verite wrote:
>
> > The patch I posted in [1] was pretty simple, but at the time, query
> > results were always discarded. Now that pgbench can instantiate
> > variables from query results, a script can do:
> > select 1 as var \gset
> > select :var;
> > This kind of sequence wouldn't work in batch mode since it
> > sends queries before getting results of previous queries.
> >
> > So maybe \gset should be rejected when inside a batch section.
>
> Hah.
>
> Hacking pgbench extensively is beyond what I'm willing to do for this
> feature at this time. Making \gset rejected in a batch section sounds
> simple enough and supports \beginbatch et al sufficiently to compare
> performance, so I'm OK with a patch that does that. That'd be a small
> extension to your previous patch, if I understand correctly.
>
> If you or others want to send patches to extend batch support with
> read-write tracking for variables, feel free, but I hereby declare that
> I'm not taking immediate responsibility for getting them committed.
I think minimal support is entirely sufficient initially.
Andres
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 14:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2020-11-18 17:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2020-11-23 21:58 ` Daniel Verite <daniel@manitou-mail.org>
2020-11-23 22:15 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
1 sibling, 1 reply; 182+ messages in thread
From: Daniel Verite @ 2020-11-23 21:58 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: David G. Johnston <david.g.johnston@gmail.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael.paquier@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; Heikki Linnakangas <heikki.linnakangas@iki.fi>
Hi,
Here's a new version with the pgbench support included.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: https://www.manitou-mail.org
Twitter: @DanielVerite
Attachments:
v26-libpq-batch.patch (text/plain, 93.4K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 9ce32fb39b..2a94f8f6b9 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3061,6 +3061,30 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-batch-end">
+ <term><literal>PGRES_BATCH_END</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents the end of a batch.
+ This status occurs only when batch mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-batch-aborted">
+ <term><literal>PGRES_BATCH_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a batch that's
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and it will return this status code,
+ until the end of the current batch, at which point it will return
+ <literal>PGRES_BATCH_END</literal> and normal processing can resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4819,6 +4843,482 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch Mode</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> batch mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the batch mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While batch mode provides a significant performance boost, writing
+ clients using the batch mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using Batch Mode</title>
+
+ <para>
+ To issue batches the application must switch a connection into batch mode.
+ Enter batch mode with <xref linkend="libpq-PQenterBatchMode"/>
+ or test whether batch mode is active with
+ <xref linkend="libpq-PQbatchStatus"/>.
+ In batch mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is not recommended as it
+ may trigger failure in batch processing. Using any synchronous
+ command execution functions such as <function>PQfn</function>,
+ <function>PQexec</function> or one of its sibling functions are error
+ conditions.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill its output buffer and the server's receive
+ buffer before switching to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ <para>
+ Batch mode consumes more memory when send/receive is not done as required,
+ even in non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests using
+ normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a
+ <xref linkend="libpq-PQbatchSendQueue"/>
+ call to mark the end of the batch. The client needs not
+ call <function>PQgetResult</function> immediately after
+ dispatching each operation.
+ <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server may begin executing the batch before all
+ commands in the batch are queued and the end of batch command is sent.
+ If any statement encounters an error the server aborts the current
+ transaction and skips processing the rest of the batch.
+ Query processing resumes after the end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one batch query, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next batch query may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the batch have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_BATCH_END</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ batch has been sent, or interleave that with sending further batch
+ queries; see <xref linkend="libpq-batch-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_BATCH_END</literal>
+ and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream and at no other time.
+ <literal>PGRES_BATCH_ABORTED</literal> is emitted during error handling;
+ see <xref linkend="libpq-batch-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries and the expected results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ still to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from
+ socket should be read frequently and there's no need to wait till the
+ batch end to read the results. Batches should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per batch.
+ There's no need to exit batch mode and re-enter it between batches
+ or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending Batch Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and
+ the end batch result has been consumed the application may return
+ to non-batched mode with <xref linkend="libpq-PQexitBatchMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-batch-functions">
+ <title>Functions Associated with Batch Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application>
+ connection.
+<synopsis>
+int PQbatchStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQbatchStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ batch mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in batch mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in aborted
+ batch status. The aborted flag is cleared as soon as the result
+ of the <function>PQbatchSendQueue</function> at the end of the aborted
+ batch is processed. Clients don't usually need this function to
+ verify aborted status, as they can tell that the batch is aborted
+ from the <literal>PGRES_BATCH_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ batch mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-batch-errors"/>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ batch mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ is failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-batch-tips">
+ <title>When to Use Batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks, but
+ pipelining can sometimes offer considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid sequence. There is usually less benefit
+ in using batches when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without batching; with batching it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ must introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Batching is less useful, and more complex, when a single batch contains
+ multiple transactions (see <xref linkend="libpq-batch-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 14.0, but clients using
+ the PostgreSQL 14 version of <application>libpq</application> can use
+ batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4859,6 +5359,17 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <!-- XXX Isn't this note merely repeating what was said above?
+ I think it should be removed. -->
+ <note>
+ <para>
+ When using batch mode, activate the single-row mode on the current
+ batch query by calling <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6329cf0796..49be8b1dbe 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 7180fedd65..921e5dccd9 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1058,6 +1058,20 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-batch'>
+ <term><literal>\beginbatch</literal></term>
+ <term><literal>\endbatch</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a batch of SQL statements.
+ In a batch, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-batch-mode"/>).
+ Batching requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-gset'>
<term>
<literal>\gset [<replaceable>prefix</replaceable>]</literal>
@@ -1086,6 +1100,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ inside a batch section, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 24f8b3e42e..9ae67387a5 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1026,6 +1026,9 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->status = WALRCV_ERROR;
walres->err = pchomp(PQerrorMessage(conn->streamConn));
break;
+ default:
+ /* This is just to keep compiler quiet */
+ break;
}
PQclear(pgres);
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 3057665bbe..5846e9153e 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,7 +351,8 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
+ * CSTATE_WAIT_RESULT state unless in batch mode.
+ * On a \sleep meta-command, the timer is set,
* and we enter the CSTATE_SLEEP state to wait for it to expire. Other
* meta-commands are executed immediately. If the command about to start
* is actually beyond the end of the script, advance to CSTATE_END_TX.
@@ -485,7 +486,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_BEGINBATCH, /* \beginbatch */
+ META_ENDBATCH /* \endbatch */
} MetaCommand;
typedef enum QueryMode
@@ -2492,6 +2495,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "beginbatch") == 0)
+ mc = META_BEGINBATCH;
+ else if (pg_strcasecmp(cmd, "endbatch") == 0)
+ mc = META_ENDBATCH;
else
mc = META_NONE;
return mc;
@@ -2681,11 +2688,24 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In batch mode, we use asynchronous functions. If a server-side
+ * error occurs, it will be processed later among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2799,6 +2819,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_BATCH_END:
+ pg_log_debug("client %d batch ending", st->id);
+ if (PQexitBatchMode(st->con) != 1)
+ pg_log_error("client %d failed to exit batch mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3057,13 +3083,27 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ if ((command->meta == META_GSET || command->meta == META_ASET)
+ && PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "SQL", "\\gset and \\aset are not allowed in a batch section");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in batch mode */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3184,6 +3224,7 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
}
break;
+
/*
* Wait for the current SQL command to complete
*/
@@ -3203,7 +3244,14 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of batch mode: stop reading results.
+ * batch mode: continue reading results until an end of batch response.
+ */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3447,6 +3495,45 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_BEGINBATCH)
+ {
+ /*
+ * In batch mode, we use a workflow based on libpq batch
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "beginbatch", "cannot use batch mode with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ return CSTATE_ABORTED;
+ }
+
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "beginbatch", "already in batch mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterBatchMode(st->con) == 0)
+ {
+ commandFailed(st, "beginbatch", "failed to start a batch");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDBATCH)
+ {
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ commandFailed(st, "endbatch", "not in batch mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQbatchSendQueue(st->con))
+ {
+ commandFailed(st, "endbatch", "failed to end the batch");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of batch mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4686,6 +4773,12 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
}
+ else if (my_command->meta == META_BEGINBATCH || my_command->meta == META_ENDBATCH)
+ {
+ if (my_command->argc != 1)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "unexpected argument", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..ca86f55652 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterBatchMode 180
+PQexitBatchMode 181
+PQbatchSendQueue 182
+PQbatchStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index e7781d010f..c673bc405f 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -536,6 +536,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -555,6 +572,7 @@ pqDropServerData(PGconn *conn)
{
PGnotify *notify;
pgParameterStatus *pstatus;
+ PGcommandQueueEntry *queue;
/* Forget pending notifies */
notify = conn->notifyHead;
@@ -567,6 +585,14 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ pqFreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ pqFreeCommandQueue(queue);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -6699,6 +6725,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+int
+PQbatchStatus(const PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index eea0237c3a..93971857d5 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -70,6 +72,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static int pqBatchProcessQueue(PGconn *conn);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
@@ -1210,7 +1217,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1233,6 +1240,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return 0;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1263,10 +1277,11 @@ PQsendQuery(PGconn *conn, const char *query)
conn->last_query = strdup(query);
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in batch mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
{
/* error message should be set up already */
return 0;
@@ -1331,6 +1346,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1362,6 +1381,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc batch memory before doing anything */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', false, conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1389,31 +1417,47 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in batch mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1461,7 +1505,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1479,20 +1524,62 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1516,6 +1603,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1525,6 +1616,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1637,35 +1745,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in batch mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1766,14 +1882,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In batch mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next batched
+ * query is being processed. At the end of the batch, returns a
+ * end-of-batch result with PQresultStatus(result) == PGRES_BATCH_END.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1842,9 +1961,38 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, we prepare the processing of the results of
+ * the next query.
+ */
+ pqBatchProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as
+ * there can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -2025,6 +2173,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2219,6 +2374,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2234,6 +2392,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2242,32 +2412,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in batch mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2665,6 +2843,13 @@ PQfn(PGconn *conn,
/* clear the error string */
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2685,6 +2870,377 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Batch mode support ======== */
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of
+ * batched commands may be sent while in batch mode. Batch mode can be exited
+ * by calling PQexitBatchMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in batch mode */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot enter batch mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Returns 1 in success (batch mode was ended, or not in batch mode).
+ *
+ * Returns 0 if in batch mode and cannot be ended yet.
+ * Error message will be set.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot exit batch mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot exit batch mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("command queue not clean"));
+ return 0;
+ }
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * internal function pqBatchProcessQueue
+ *
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+static int
+pqBatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 0;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ printfPQExpBuffer(&conn->errorMessage,
+ "internal error, COPY in batch mode");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ /* should be unreachable */
+ printfPQExpBuffer(&conn->errorMessage,
+ "internal error, IDLE in batch mode");
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQexitBatchMode() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode XXX why?? */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result =
+ PQmakeEmptyPGresult(conn, PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
+/*
+ * PQbatchSendQueue
+ * End a batch submission.
+ *
+ * It's legal to start submitting another batch immediately, without
+ * waiting for the results of the current batch. There's no need to end batch
+ * mode and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and
+ * including the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED
+ * state. If the whole batch is processed without error, a PGresult with
+ * PGRES_BATCH_END is produced.
+ *
+ * Queries can already have been sent before PQbatchSendQueue is called, but
+ * PQbatchSendQueue need to be called before retrieving command results.
+ *
+ * The connection will remain in batch mode and unavailable for new synchronous
+ * command execution functions until all results from the batch are processed
+ * by the client.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot send batch when not in batch mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ /* should be unreachable */
+ printfPQExpBuffer(&conn->errorMessage,
+ "internal error: cannot send batch when idle\n");
+ return 0;
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ printfPQExpBuffer(&conn->errorMessage,
+ "internal error: cannot send batch while in COPY\n");
+ return 0;
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqBatchProcessQueue so the user can call start calling getResult.
+ */
+ pqBatchProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3285,6 +3841,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqBatchFlush
+ *
+ * In batch mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-batch mode, data will be flushed all the time.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return (pqFlush(conn));
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 9360c541be..2ff3fa4883 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -406,6 +406,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 1696525475..da38e6aed1 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -217,10 +217,19 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -875,6 +884,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 3b6a9fbce3..fcdd887958 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -97,7 +97,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -137,6 +140,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -328,6 +342,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern int PQbatchStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -435,6 +450,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1de91ae295..d2b26f2299 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -394,6 +416,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -406,6 +429,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -798,6 +831,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index a6d2ffbf9e..287214c544 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..11e8463984
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..6d3a0ea4d5
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/test_libpq/Makefile
+
+PROGRAM = testlibpqbatch
+OBJS = testlibpqbatch.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..0b27896a2a
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_batch simple_batch multi_batch batch_abort
+ batch_insert singlerow);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'testlibpqbatch', $testname, $node->connstr('postgres'), $numrows ],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000000..7157e6cb90
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,963 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ * Verify libpq batch execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "testlibpqbatch";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("Unable to enter batch mode\n");
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Batch mode not activated properly\n");
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in batch mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in batch mode but succeeded\n");
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("re-entering batch mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in batch, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty batch mode\n");
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ pg_fatal("Batch mode not terminated properly\n");
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("batch mode exit when not in batch mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitBatchMode(conn) != 0)
+ pg_fatal("exiting batch mode with work in progress should fail, but succeeded\n");
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Ending a batch failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a batch item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQexitBatchMode(conn) != 0)
+ pg_fatal("exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_BATCH_END expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after end batch call\n");
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("attempt to exit batch mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ pg_fatal("Exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the batch results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a batch item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitBatchMode(conn) != 0)
+ pg_fatal("exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* second batch */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a batch item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a batch item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("attempt to exit batch mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ pg_fatal("exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Sending first batch failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-batch insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a batch aborted message for the second insert, a batch-end, then a
+ * command-ok and a batch-ok for the second batch operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ pg_fatal("batch should be flagged as aborted but isn't\n");
+
+ /* third query in batch, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_BATCH_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ pg_fatal("batch should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in batch */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /*
+ * The end of a failed batch is a PGRES_BATCH_END.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code from first batch sync\n"
+ "Expected PGRES_BATCH_END, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* XXX why do we have a NULL result after PGRES_BATCH_END? */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /* the insert from the second batch */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second batch\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second batch sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code %s from second batch sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("attempt to exit batch mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ pg_fatal("exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd batch
+ * - insert applied from 2nd batch
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_batch_insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+test_batch_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSendQueue(conn) == 1)
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("attempt to exit batch mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool batch_ended = false;
+
+ /* 1 batch, 3 queries in it */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQbatchSendQueue(conn);
+
+ for (i = 0; !batch_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, "end of batch reached\n");
+ batch_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!batch_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("failed to end batch mode: %s\n", PQerrorMessage(conn));
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's batch-mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_batch|simple_batch|multi_batch|batch_abort|\n");
+ fprintf(stderr, " singlerow|batch_insert\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (strcmp(argv[1], "disallowed_in_batch") == 0)
+ test_disallowed_in_batch(conn);
+ else if (strcmp(argv[1], "simple_batch") == 0)
+ test_simple_batch(conn);
+ else if (strcmp(argv[1], "multi_batch") == 0)
+ test_multi_batch(conn);
+ else if (strcmp(argv[1], "batch_abort") == 0)
+ test_batch_abort(conn);
+ else if (strcmp(argv[1], "batch_insert") == 0)
+ test_batch_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 90594bd41b..634e48ec85 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -50,7 +50,8 @@ my @contrib_excludes = (
'pgcrypto', 'sepgsql',
'brin', 'test_extensions',
'test_misc', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'snapshot_too_old', 'unsafe_tests',
+ 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 03c4e0fe5b..9b75db962b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -215,6 +215,7 @@ BackgroundWorkerHandle
BackgroundWorkerSlot
Barrier
BaseBackupCmd
+BatchInsertStep
BeginDirectModify_function
BeginForeignInsert_function
BeginForeignModify_function
@@ -1544,6 +1545,7 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
@@ -1656,6 +1658,7 @@ PMSignalReason
PMState
POLYGON
PQArgBlock
+PQBatchStatus
PQEnvironmentOption
PQExpBuffer
PQExpBufferData
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 14:42 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
2020-11-18 17:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-23 21:58 ` Re: PATCH: Batch/pipelining support for libpq Daniel Verite <daniel@manitou-mail.org>
@ 2020-11-23 22:15 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 0 replies; 182+ messages in thread
From: Alvaro Herrera @ 2020-11-23 22:15 UTC (permalink / raw)
To: Daniel Verite <daniel@manitou-mail.org>; +Cc: David G. Johnston <david.g.johnston@gmail.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael.paquier@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; Heikki Linnakangas <heikki.linnakangas@iki.fi>
On 2020-Nov-23, Daniel Verite wrote:
> Hi,
>
> Here's a new version with the pgbench support included.
Thanks, incorporated into my local copy.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2020-11-16 23:29 ` David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
1 sibling, 1 reply; 182+ messages in thread
From: David G. Johnston @ 2020-11-16 23:29 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Daniel Verite <daniel@manitou-mail.org>; Michael Paquier <michael.paquier@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; Heikki Linnakangas <heikki.linnakangas@iki.fi>
On Fri, Nov 13, 2020 at 5:38 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:
> Here's a v25.
>
> I made a few more changes to the docs per David's suggestions; I also
> reordered the sections quite a bit. It's now like this:
> * Batch Mode
> * Using Batch Mode
> * Issuing Queries
> * Processing Results
> * Error Handling
> * Interleaving Result Processing and Query Dispatch
> * Ending Batch Mode
> * Functions Associated with Batch Mode
> * When to Use Batching
>
Thanks!
I like the new flow and changes. I've attached a patch that covers some
missing commas and typos along with a few parts that made me pause.
The impact of memory came out of nowhere under the non-blocking mode
commentary. I took a stab at incorporating it more broadly.
The "are error conditions" might be a well-known phrasing to those using
libpq but that sentence reads odd to me. I did try to make the example
listing flow a bit better and added a needed comma.
Tried to clean up a few phrasings after that. The error handling part I'm
not especially happy with but I think it's closer and more useful than just
"emitted during error handling" - it gets emitted upon error, handling is a
client concern.
Seems odd to say the new feature was introduced in v14.0, the .0 seems ok
to imply. I didn't actually fix it in the attached but "the PostgreSQL 14
version of libpq" is going to become outdated quickly, better just to drop
it.
"The batch API was introduced in PostgreSQL 14, but clients can use batches
on servers supporting v3 of the extended query protocol, potentially going
as far back as version 7.4."
David J.
Attachments:
v25-01-libpq-batch-dgj-doc-suggestions.patch (application/octet-stream, 7.3K) download
commit a8b461c0333db905ee4f6a9d2bfc657a37543893
Author: David G. Johnston <david.g.johnston@gmail.com>
Date: Mon Nov 16 23:17:10 2020 +0000
v25 doc suggestions
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 2a94f8f6b9..f446c013ea 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4871,6 +4871,13 @@ int PQflush(PGconn *conn);
corresponds to which query in the queue.
</para>
+ <para>
+ Batch mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
<sect2 id="libpq-batch-using">
<title>Using Batch Mode</title>
@@ -4882,9 +4889,9 @@ int PQflush(PGconn *conn);
In batch mode, only <link linkend="libpq-async">asynchronous operations</link>
are permitted, and <literal>COPY</literal> is not recommended as it
may trigger failure in batch processing. Using any synchronous
- command execution functions such as <function>PQfn</function>,
- <function>PQexec</function> or one of its sibling functions are error
- conditions.
+ command execution functions, such as <function>PQfn</function>, or
+ <function>PQexec</function> and its sibling functions, is an error
+ condition.
</para>
<note>
@@ -4903,10 +4910,6 @@ int PQflush(PGconn *conn);
</para>
</footnote>
</para>
- <para>
- Batch mode consumes more memory when send/receive is not done as required,
- even in non-blocking mode.
- </para>
</note>
<sect3 id="libpq-batch-sending">
@@ -4914,13 +4917,13 @@ int PQflush(PGconn *conn);
<para>
After entering batch mode the application dispatches requests using
- normal asynchronous <application>libpq</application> functions such as
+ normal asynchronous <application>libpq</application> functions, such as:
<function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
<function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
- <function>PQsendDescribePrepared</function>.
+ and <function>PQsendDescribePrepared</function>.
The asynchronous requests are followed by a
<xref linkend="libpq-PQbatchSendQueue"/>
- call to mark the end of the batch. The client needs not
+ call to mark the end of the batch. The client need not
call <function>PQgetResult</function> immediately after
dispatching each operation.
<link linkend="libpq-batch-results">Result processing</link>
@@ -4929,8 +4932,8 @@ int PQflush(PGconn *conn);
<para>
The server executes statements, and returns results, in the order the
- client sends them. The server may begin executing the batch before all
- commands in the batch are queued and the end of batch command is sent.
+ client sends them. The server will begin executing the batch commands
+ immediately, not waiting for the end of batch command.
If any statement encounters an error the server aborts the current
transaction and skips processing the rest of the batch.
Query processing resumes after the end of the failed batch.
@@ -4981,8 +4984,10 @@ int PQflush(PGconn *conn);
and <literal>PGRES_BATCH_ABORTED</literal>.
<literal>PGRES_BATCH_END</literal> is reported exactly once for each
<function>PQbatchSendQueue</function> call at the corresponding point in
- the result stream and at no other time.
- <literal>PGRES_BATCH_ABORTED</literal> is emitted during error handling;
+ the result stream.
+ <literal>PGRES_BATCH_ABORTED</literal> is emitted in place of a normal
+ result stream result for the first error and all subsequent results
+ except <literal>PGRES_BATCH_END</literal> and null;
see <xref linkend="libpq-batch-errors"/>.
</para>
@@ -5038,8 +5043,8 @@ int PQflush(PGconn *conn);
transaction state at the end of the batch. If a batch contains <emphasis>
multiple explicit transactions</emphasis>, all transactions that committed
prior to the error remain committed, the currently in-progress transaction
- is aborted and all subsequent operations in the current and all later
- transactions in the same batch are skipped completely.
+ is aborted, and all subsequent operations are skipped completely. Including
+ subsequent transactions.
</para>
<note>
@@ -5071,10 +5076,10 @@ int PQflush(PGconn *conn);
it should dispatch more work. When the socket is readable it should
read results and process them, matching them up to the next entry in
its expected results queue. Based on available memory, results from
- socket should be read frequently and there's no need to wait till the
+ socket should be read frequently: there's no need to wait until the
batch end to read the results. Batches should be scoped to logical
units of work, usually (but not necessarily) one transaction per batch.
- There's no need to exit batch mode and re-enter it between batches
+ There's no need to exit batch mode and re-enter it between batches,
or to wait for one batch to finish before sending the next.
</para>
@@ -5090,8 +5095,8 @@ int PQflush(PGconn *conn);
<title>Ending Batch Mode</title>
<para>
- Once all dispatched commands have had their results processed and
- the end batch result has been consumed the application may return
+ Once all dispatched commands have had their results processed, and
+ the end batch result has been consumed, the application may return
to non-batched mode with <xref linkend="libpq-PQexitBatchMode"/>.
</para>
</sect3>
@@ -5258,10 +5263,11 @@ int PQbatchSendQueue(PGconn *conn);
<title>When to Use Batching</title>
<para>
- Much like asynchronous query mode, there is no performance disadvantage to
- using batching and pipelining. It increases client application complexity
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using batching. It increases client application complexity,
and extra caution is required to prevent client/server deadlocks, but
- pipelining can sometimes offer considerable performance improvements.
+ batching can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
</para>
<para>
@@ -5279,7 +5285,7 @@ int PQbatchSendQueue(PGconn *conn);
Use batches when your application does lots of small
<literal>INSERT</literal>, <literal>UPDATE</literal> and
<literal>DELETE</literal> operations that can't easily be transformed
- into operations on sets or into a <literal>COPY</literal> operation.
+ into operations on sets, or into a <literal>COPY</literal> operation.
</para>
<para>
@@ -5310,7 +5316,7 @@ UPDATE mytable SET x = x + 1 WHERE id = 42;
<note>
<para>
- The batch API was introduced in PostgreSQL 14.0, but clients using
+ The batch API was introduced in PostgreSQL 14, but clients using
the PostgreSQL 14 version of <application>libpq</application> can use
batches on server versions 7.4 and newer. Batching works on any server
that supports the v3 extended query protocol.
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
@ 2021-01-21 23:39 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-22 00:14 ` Re: PATCH: Batch/pipelining support for libpq Zhihong Yu <zyu@yugabyte.com>
0 siblings, 2 replies; 182+ messages in thread
From: Alvaro Herrera @ 2021-01-21 23:39 UTC (permalink / raw)
To: David G. Johnston <david.g.johnston@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael.paquier@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; Heikki Linnakangas <heikki.linnakangas@iki.fi>
Thanks David Johnston and Daniel Vérité, I have incorporated your
changes into this patch, which is now v26. Also, it's been rebased on
current sources.
I've been using the new PQtrace() stuff to verify the behavior of the
new feature. It's not perfect, but at least it doesn't crash
immediately as it did when I tried a few weeks ago. There are
imperfections that I think are due to bugs in the PQtrace
implementation, not in this patch.
--
Álvaro Herrera 39°49'30"S 73°17'W
"El conflicto es el camino real hacia la unión"
Attachments:
v26-libpq-batch.patch (text/x-diff, 92.7K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 2bb3bf77e4..d22b866677 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3099,6 +3099,30 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-batch-end">
+ <term><literal>PGRES_BATCH_END</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents the end of a batch.
+ This status occurs only when batch mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-batch-aborted">
+ <term><literal>PGRES_BATCH_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a batch that's
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and it will return this status code,
+ until the end of the current batch, at which point it will return
+ <literal>PGRES_BATCH_END</literal> and normal processing can resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4857,6 +4881,488 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch Mode</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> batch mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the batch mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While batch mode provides a significant performance boost, writing
+ clients using the batch mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Batch mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using Batch Mode</title>
+
+ <para>
+ To issue batches the application must switch a connection into batch mode.
+ Enter batch mode with <xref linkend="libpq-PQenterBatchMode"/>
+ or test whether batch mode is active with
+ <xref linkend="libpq-PQbatchStatus"/>.
+ In batch mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is not recommended as it
+ may trigger failure in batch processing. Using any synchronous
+ command execution functions, such as <function>PQfn</function>, or
+ <function>PQexec</function> and its sibling functions, is an error
+ condition.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill its output buffer and the server's receive
+ buffer before switching to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests using
+ normal asynchronous <application>libpq</application> functions, such as:
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ and <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a
+ <xref linkend="libpq-PQbatchSendQueue"/>
+ call to mark the end of the batch. The client need not
+ call <function>PQgetResult</function> immediately after
+ dispatching each operation.
+ <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the batch commands
+ immediately, not waiting for the end of the batch.
+ If any statement encounters an error the server aborts the current
+ transaction and skips processing the rest of the batch.
+ Query processing resumes after the end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one batch query, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next batch query may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the batch have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_BATCH_END</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ batch has been sent, or interleave that with sending further batch
+ queries; see <xref linkend="libpq-batch-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_BATCH_END</literal>
+ and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchSendQueue</function> call at the corresponding point in
+ the result stream.
+ <literal>PGRES_BATCH_ABORTED</literal> is emitted in place of a normal
+ result stream result for the first error and all subsequent results
+ except <literal>PGRES_BATCH_END</literal> and null;
+ see <xref linkend="libpq-batch-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries and the expected results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchSendQueue</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ still to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from
+ socket should be read frequently: there's no need to wait until the
+ batch end to read the results. Batches should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per batch.
+ There's no need to exit batch mode and re-enter it between batches,
+ or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending Batch Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed, and
+ the end batch result has been consumed, the application may return
+ to non-batched mode with <xref linkend="libpq-PQexitBatchMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-batch-functions">
+ <title>Functions Associated with Batch Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application>
+ connection.
+<synopsis>
+int PQbatchStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQbatchStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ batch mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in batch mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in aborted
+ batch status. The aborted flag is cleared as soon as the result
+ of the <function>PQbatchSendQueue</function> at the end of the aborted
+ batch is processed. Clients don't usually need this function to
+ verify aborted status, as they can tell that the batch is aborted
+ from the <literal>PGRES_BATCH_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterBatchMode">
+ <term>
+ <function>PQenterBatchMode</function>
+ <indexterm>
+ <primary>PQenterBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQenterBatchMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitBatchMode">
+ <term>
+ <function>PQexitBatchMode</function>
+ <indexterm>
+ <primary>PQexitBatchMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitBatchMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ batch mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchSendQueue">
+ <term>
+ <function>PQbatchSendQueue</function>
+ <indexterm>
+ <primary>PQbatchSendQueue</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-batch-errors"/>.
+
+<synopsis>
+int PQbatchSendQueue(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ batch mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ is failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-batch-tips">
+ <title>When to Use Batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using batching. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ batching can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid sequence. There is usually less benefit
+ in using batches when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without batching; with batching it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Batching is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ must introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Batching is less useful, and more complex, when a single batch contains
+ multiple transactions (see <xref linkend="libpq-batch-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 14, but clients using
+ the PostgreSQL 14 version of <application>libpq</application> can use
+ batches on server versions 7.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4897,6 +5403,17 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <!-- XXX Isn't this note merely repeating what was said above?
+ I think it should be removed. -->
+ <note>
+ <para>
+ When using batch mode, activate the single-row mode on the current
+ batch query by calling <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ See <xref linkend="libpq-batch-mode"/> for more information.
+ </para>
+ </note>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 413eda50af..422990d66f 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while libpq connection is in batch mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index faa7c26b0a..9ff5d774f3 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1068,6 +1068,20 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-batch'>
+ <term><literal>\beginbatch</literal></term>
+ <term><literal>\endbatch</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a batch of SQL statements.
+ In a batch, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-batch-mode"/>).
+ Batching requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-gset'>
<term>
<literal>\gset [<replaceable>prefix</replaceable>]</literal>
@@ -1096,6 +1110,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ inside a batch section, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f7da3e1f62..3d8204f0ed 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,7 +351,8 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
+ * CSTATE_WAIT_RESULT state unless in batch mode.
+ * On a \sleep meta-command, the timer is set,
* and we enter the CSTATE_SLEEP state to wait for it to expire. Other
* meta-commands are executed immediately. If the command about to start
* is actually beyond the end of the script, advance to CSTATE_END_TX.
@@ -485,7 +486,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_BEGINBATCH, /* \beginbatch */
+ META_ENDBATCH /* \endbatch */
} MetaCommand;
typedef enum QueryMode
@@ -2506,6 +2509,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "beginbatch") == 0)
+ mc = META_BEGINBATCH;
+ else if (pg_strcasecmp(cmd, "endbatch") == 0)
+ mc = META_ENDBATCH;
else
mc = META_NONE;
return mc;
@@ -2695,11 +2702,24 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In batch mode, we use asynchronous functions. If a server-side
+ * error occurs, it will be processed later among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2813,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_BATCH_END:
+ pg_log_debug("client %d batch ending", st->id);
+ if (PQexitBatchMode(st->con) != 1)
+ pg_log_error("client %d failed to exit batch mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3071,13 +3097,27 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ if ((command->meta == META_GSET || command->meta == META_ASET)
+ && PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "SQL", "\\gset and \\aset are not allowed in a batch section");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in batch mode */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3217,7 +3257,14 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of batch mode: stop reading results.
+ * batch mode: continue reading results until an end of batch response.
+ */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3461,6 +3508,45 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_BEGINBATCH)
+ {
+ /*
+ * In batch mode, we use a workflow based on libpq batch
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "beginbatch", "cannot use batch mode with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ return CSTATE_ABORTED;
+ }
+
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "beginbatch", "already in batch mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterBatchMode(st->con) == 0)
+ {
+ commandFailed(st, "beginbatch", "failed to start a batch");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDBATCH)
+ {
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ commandFailed(st, "endbatch", "not in batch mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQbatchSendQueue(st->con))
+ {
+ commandFailed(st, "endbatch", "failed to end the batch");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of batch mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4700,6 +4786,12 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
}
+ else if (my_command->meta == META_BEGINBATCH || my_command->meta == META_ENDBATCH)
+ {
+ if (my_command->argc != 1)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "unexpected argument", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..ca86f55652 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterBatchMode 180
+PQexitBatchMode 181
+PQbatchSendQueue 182
+PQbatchStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 8ca0583aa9..150b4c6744 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -536,6 +536,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -567,6 +584,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -6656,6 +6679,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+int
+PQbatchStatus(const PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->batch_status;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index e730753387..9a20793b48 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static int pqBatchProcessQueue(PGconn *conn);
+static int pqBatchFlush(PGconn *conn);
/* ----------------
@@ -1160,7 +1167,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1186,6 +1193,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, true);
}
@@ -1228,10 +1242,11 @@ PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
conn->last_query = strdup(query);
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in batch mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
{
/* error message should be set up already */
return 0;
@@ -1296,6 +1311,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1327,6 +1346,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc batch memory before doing anything */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', false, conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1354,31 +1382,47 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in batch mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1426,7 +1470,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1447,20 +1492,62 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->batch_status == PQBATCH_MODE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1484,6 +1571,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1493,6 +1584,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1605,35 +1713,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in batch mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1734,14 +1850,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In batch mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next batched
+ * query is being processed. At the end of the batch, returns a
+ * end-of-batch result with PQresultStatus(result) == PGRES_BATCH_END.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1810,9 +1929,38 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, we prepare the processing of the results of
+ * the next query.
+ */
+ pqBatchProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * In batch mode, query execution state cannot be IDLE as
+ * there can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1993,6 +2141,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2177,6 +2332,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2192,6 +2350,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2200,32 +2370,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in batch mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqBatchFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2624,6 +2802,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in batch mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2644,6 +2829,372 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Batch mode support ======== */
+
+/*
+ * PQenterBatchMode
+ * Put an idle connection in batch mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of
+ * batched commands may be sent while in batch mode. Batch mode can be exited
+ * by calling PQexitBatchMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in batch mode */
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter batch mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitBatchMode
+ * End batch mode and return to normal command mode.
+ *
+ * Returns 1 in success (batch mode was ended, or not in batch mode).
+ *
+ * Returns 0 if in batch mode and cannot be ended yet.
+ * Error message will be set.
+ */
+int
+PQexitBatchMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit batch mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit batch mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("command queue not clean"));
+ return 0;
+ }
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * internal function pqBatchProcessQueue
+ *
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns 1 if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns 0 if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+static int
+pqBatchProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->batch_status != PQBATCH_MODE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error, COPY in batch mode");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return 0;
+ break;
+ case PGASYNC_IDLE:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error, IDLE in batch mode");
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQexitBatchMode() now.
+ */
+ return 0;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode XXX why?? */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result =
+ PQmakeEmptyPGresult(conn, PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ return 0;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return 1;
+}
+
+/*
+ * PQbatchSendQueue
+ * End a batch submission.
+ *
+ * It's legal to start submitting another batch immediately, without
+ * waiting for the results of the current batch. There's no need to end batch
+ * mode and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and
+ * including the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED
+ * state. If the whole batch is processed without error, a PGresult with
+ * PGRES_BATCH_END is produced.
+ *
+ * Queries can already have been sent before PQbatchSendQueue is called, but
+ * PQbatchSendQueue need to be called before retrieving command results.
+ *
+ * The connection will remain in batch mode and unavailable for new synchronous
+ * command execution functions until all results from the batch are processed
+ * by the client.
+ */
+int
+PQbatchSendQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send batch when not in batch mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send batch when idle\n");
+ return 0;
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send batch while in COPY\n");
+ return 0;
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqBatchProcessQueue so the user can call start calling getResult.
+ */
+ pqBatchProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3248,6 +3799,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqBatchFlush
+ *
+ * In batch mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-batch mode, data will be flushed all the time.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqBatchFlush(PGconn *conn)
+{
+ if ((conn->batch_status == PQBATCH_MODE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return (pqFlush(conn));
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 6efa53d8b7..f034676e95 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -406,6 +406,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index e4ee9d69d2..9d66514da1 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -217,10 +217,19 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -875,6 +884,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index c266ad5b13..634d832164 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -97,7 +97,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -137,6 +140,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -328,6 +342,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern int PQbatchStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -435,6 +450,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQenterBatchMode(PGconn *conn);
+extern int PQexitBatchMode(PGconn *conn);
+extern int PQbatchSendQueue(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 4db498369c..5d6388e5bf 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -394,6 +416,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -406,6 +429,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -802,6 +835,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 59921b46cf..3dc3b184dc 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..11e8463984
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..6d3a0ea4d5
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/test_libpq/Makefile
+
+PROGRAM = testlibpqbatch
+OBJS = testlibpqbatch.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..0b27896a2a
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_batch simple_batch multi_batch batch_abort
+ batch_insert singlerow);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'testlibpqbatch', $testname, $node->connstr('postgres'), $numrows ],
+ "testlibpqbatch $testname");
+}
+
+$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000000..7157e6cb90
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,963 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ * Verify libpq batch execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "testlibpqbatch";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("Unable to enter batch mode\n");
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Batch mode not activated properly\n");
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in batch mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in batch mode but succeeded\n");
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("re-entering batch mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in batch, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty batch mode\n");
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ pg_fatal("Batch mode not terminated properly\n");
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("batch mode exit when not in batch mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitBatchMode(conn) != 0)
+ pg_fatal("exiting batch mode with work in progress should fail, but succeeded\n");
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Ending a batch failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a batch item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQexitBatchMode(conn) != 0)
+ pg_fatal("exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_BATCH_END expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after end batch call\n");
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("attempt to exit batch mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ pg_fatal("Exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Ending first batch failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the batch results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a batch item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitBatchMode(conn) != 0)
+ pg_fatal("exiting batch mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* second batch */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a batch item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a batch item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("attempt to exit batch mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ pg_fatal("exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Sending first batch failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-batch insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQbatchSendQueue(conn) != 1)
+ pg_fatal("Ending second batch failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a batch aborted message for the second insert, a batch-end, then a
+ * command-ok and a batch-ok for the second batch operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ pg_fatal("batch should be flagged as aborted but isn't\n");
+
+ /* third query in batch, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_BATCH_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ pg_fatal("batch should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in batch */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /*
+ * The end of a failed batch is a PGRES_BATCH_END.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code from first batch sync\n"
+ "Expected PGRES_BATCH_END, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* XXX why do we have a NULL result after PGRES_BATCH_END? */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /* the insert from the second batch */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second batch\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second batch sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ pg_fatal("Unexpected result code %s from second batch sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ pg_fatal("Fell out of batch mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("attempt to exit batch mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ pg_fatal("exiting batch mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd batch
+ * - insert applied from 2nd batch
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_batch_insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+test_batch_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchSendQueue(conn) == 1)
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("attempt to exit batch mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool batch_ended = false;
+
+ /* 1 batch, 3 queries in it */
+ if (PQenterBatchMode(conn) != 1)
+ pg_fatal("failed to enter batch mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQbatchSendQueue(conn);
+
+ for (i = 0; !batch_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_BATCH_END)
+ {
+ fprintf(stderr, "end of batch reached\n");
+ batch_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!batch_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitBatchMode(conn) != 1)
+ pg_fatal("failed to end batch mode: %s\n", PQerrorMessage(conn));
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's batch-mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_batch|simple_batch|multi_batch|batch_abort|\n");
+ fprintf(stderr, " singlerow|batch_insert\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (strcmp(argv[1], "disallowed_in_batch") == 0)
+ test_disallowed_in_batch(conn);
+ else if (strcmp(argv[1], "simple_batch") == 0)
+ test_simple_batch(conn);
+ else if (strcmp(argv[1], "multi_batch") == 0)
+ test_multi_batch(conn);
+ else if (strcmp(argv[1], "batch_abort") == 0)
+ test_batch_abort(conn);
+ else if (strcmp(argv[1], "batch_insert") == 0)
+ test_batch_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 5634b2d40c..ef21514164 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -50,7 +50,8 @@ my @contrib_excludes = (
'pgcrypto', 'sepgsql',
'brin', 'test_extensions',
'test_misc', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'snapshot_too_old', 'unsafe_tests',
+ 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 943142ced8..d60102d9d3 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -214,6 +214,7 @@ BackgroundWorkerHandle
BackgroundWorkerSlot
Barrier
BaseBackupCmd
+BatchInsertStep
BeginDirectModify_function
BeginForeignInsert_function
BeginForeignModify_function
@@ -1560,6 +1561,7 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
@@ -1672,6 +1674,7 @@ PMSignalReason
PMState
POLYGON
PQArgBlock
+PQBatchStatus
PQEnvironmentOption
PQExpBuffer
PQExpBufferData
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2021-01-21 23:43 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
1 sibling, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-01-21 23:43 UTC (permalink / raw)
To: David G. Johnston <david.g.johnston@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael.paquier@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; Heikki Linnakangas <heikki.linnakangas@iki.fi>
As you can see in an XXX comment in the libpq test program, the current
implementation has the behavior that PQgetResult() returns NULL after a
batch is finished and has reported PGRES_BATCH_END. I don't know if
there's a hard reason to do that, but I'd like to supress it because it
seems weird and out of place.
--
Álvaro Herrera 39°49'30"S 73°17'W
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2021-02-10 23:51 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-02-10 23:51 UTC (permalink / raw)
To: Craig Ringer <craig.ringer@enterprisedb.com>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
On 2021-Jan-21, Alvaro Herrera wrote:
> As you can see in an XXX comment in the libpq test program, the current
> implementation has the behavior that PQgetResult() returns NULL after a
> batch is finished and has reported PGRES_BATCH_END. I don't know if
> there's a hard reason to do that, but I'd like to supress it because it
> seems weird and out of place.
Hello Craig, a question for you since this API is your devising. I've
been looking at changing the way this works to prevent those NULL
returns from PQgetResult. That is, instead of having what seems like a
"NULL separator" of query results, you'd just get the PGRES_BATCH_END to
signify a batch end (not a NULL result after the BATCH_END); and the
normal PGRES_COMMAND_OK or PGRES_TUPLES_OK etc when the result of a
command has been sent. It's not working yet so I'm not sending an
updated patch, but I wanted to know if you had a rationale for including
this NULL return "separator" or was it just a convenience because of how
the code grew together.
Such a decision has obvious consequences for the test program (which
right now expects that PQgetResult() returns NULL at each step); and
naturally for libpq's internal state machine that controls how it all
works.
Thanks,
--
Álvaro Herrera 39°49'30"S 73°17'W
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2021-02-16 01:19 ` Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
0 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2021-02-16 01:19 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
On Thu, 11 Feb 2021 at 07:51, Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:
> On 2021-Jan-21, Alvaro Herrera wrote:
>
> > As you can see in an XXX comment in the libpq test program, the current
> > implementation has the behavior that PQgetResult() returns NULL after a
> > batch is finished and has reported PGRES_BATCH_END. I don't know if
> > there's a hard reason to do that, but I'd like to supress it because it
> > seems weird and out of place.
>
> Hello Craig, a question for you since this API is your devising. I've
> been looking at changing the way this works to prevent those NULL
> returns from PQgetResult. That is, instead of having what seems like a
> "NULL separator" of query results, you'd just get the PGRES_BATCH_END to
> signify a batch end (not a NULL result after the BATCH_END); and the
> normal PGRES_COMMAND_OK or PGRES_TUPLES_OK etc when the result of a
> command has been sent. It's not working yet so I'm not sending an
> updated patch, but I wanted to know if you had a rationale for including
> this NULL return "separator" or was it just a convenience because of how
> the code grew together.
>
The existing API for libpq actually specifies[1] that you should call
PQgetResult() until it returns NULL:
> After successfully calling PQsendQuery, call PQgetResult one or more
times to obtain the results. PQsendQuery cannot be called again (on the
same connection) until PQgetResult has returned a null pointer, indicating
that the command is done.
Similarly, in single-row mode, the existing API specifies that you should
call PQgetResult() until it returns NULL.
Also, IIRC the protocol already permits multiple result sets to be
returned, and the caller cannot safely assume that a single PQsendQuery()
will produce only a single result set. (I really should write a test
extension that exercises that and how libpq reacts to it.)
That's why I went for the NULL return. I think. It's been a while now so
it's a bit fuzzy.
I would definitely like an API that does not rely on testing for a NULL
return. Especially since NULL return can be ambiguous in the context of
row-at-a-time mode. New explicit enumerations for PGresult would make a lot
more sense.
So +1 from me for the general idea. I need to look at the patch as it has
evolved soon too.
Remember that the original patch I submitted for this was a 1-day weekend
hack and proof of concept to show that libpq could be modified to support
query pipelining (and thus batching too), so I could illustrate the
performance benefits that could be attained by doing so. I'd been aware of
the benefits and the protocol's ability to support it for some time thanks
to working on PgJDBC, but couldn't get anyone interested without some C
code to demonstrate it, so I wrote some. I am not going to argue that the
API I added for it is ideal in any way, and happy to see improvements.
The only change I would very strongly object to would be anything that
turned this into a *batch* mode without query-pipelining support. If you
have to queue all the queries up in advance then send them as a batch and
wait for all the results, you miss out on a lot of the potential
round-trip-time optimisations and you add initial latency. So long as there
is a way to "send A", "send B", "send C", "read results from A", "send D",
and there's a way for the application to associate some kind of state (an
application specific id or index, a pointer to an application query-queue
struct, whatever) so it can match queries to results ... then I'm happy.
[1]
https://www.postgresql.org/docs/current/libpq-async.html#LIBPQ-PQSENDQUERY
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
@ 2021-02-16 01:38 ` Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2021-02-16 01:38 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
On Tue, 16 Feb 2021 at 09:19, Craig Ringer <craig.ringer@enterprisedb.com>
wrote:
>
> So long as there is a way to "send A", "send B", "send C", "read results
> from A", "send D", and there's a way for the application to associate some
> kind of state (an application specific id or index, a pointer to an
> application query-queue struct, whatever) so it can match queries to
> results ... then I'm happy.
>
FWIW I'm also thinking of revising the docs to mostly use the term
"pipeline" instead of "batch". Use "pipelining and batching" in the chapter
topic, and mention "batch" in the index, and add a para that explains how
to run batches on top of pipelining, but otherwise use the term "pipeline"
not "batch".
That's because pipelining isn't actually batching, and using it as a naïve
batch interface will get you in trouble. If you PQsendQuery(...) a long
list of queries without consuming results, you'll block unless you're in
libpq's nonblocking-send mode. You'll then be deadlocked because the server
can't send results until you consume some (tx buffer full) and can't
consume queries until it can send some results, but you can't consume
results because you're blocked on a send that'll never end.
An actual batch interface where you can bind and send a long list of
queries might be worthwhile, but should be addressed separately, and it'd
be confusing if this pipelining interface claimed the term "batch". I'm not
convinced enough application developers actually code directly against
libpq for it to be worth creating a pretty batch interface where you can
submit (say) an array of struct PQbatchEntry { const char * query, int
nparams, ... } to a PQsendBatch(...) and let libpq handle the socket I/O
for you. But if someone wants to add one later it'll be easier if we don't
use the term "batch" for the pipelining feature.
I'll submit a reworded patch in a bit.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
@ 2021-02-16 16:27 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-02-16 16:27 UTC (permalink / raw)
To: Craig Ringer <craig.ringer@enterprisedb.com>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
On 2021-Feb-16, Craig Ringer wrote:
> FWIW I'm also thinking of revising the docs to mostly use the term
> "pipeline" instead of "batch". Use "pipelining and batching" in the chapter
> topic, and mention "batch" in the index, and add a para that explains how
> to run batches on top of pipelining, but otherwise use the term "pipeline"
> not "batch".
Hmm, this is a good point. It means I have a lot of API renaming to do.
I'll get on it now and come back with a proposal.
--
Álvaro Herrera Valdivia, Chile
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2021-02-16 23:13 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:45 ` Re: PATCH: Batch/pipelining support for libpq Zhihong Yu <zyu@yugabyte.com>
2021-02-20 00:23 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-03-03 07:07 ` RE: PATCH: Batch/pipelining support for libpq k.jamison@fujitsu.com <k.jamison@fujitsu.com>
0 siblings, 3 replies; 182+ messages in thread
From: Alvaro Herrera @ 2021-02-16 23:13 UTC (permalink / raw)
To: Craig Ringer <craig.ringer@enterprisedb.com>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
Here's a new version, where I've renamed everything to "pipeline". I
think the docs could use some additional tweaks now in order to make a
coherent story on pipeline mode, how it can be used in a batched
fashion, etc.
Here's the renames I applied. It's mostly mechanical, except
PQbatchSendQueue is now PQsendPipeline:
PQBatchStatus -> PGpipelineStatus (enum)
PQBATCH_MODE_OFF -> PQ_PIPELINE_OFF
PQBATCH_MODE_ON -> PQ_PIPELINE_ON
PQBATCH_MODE_ABORTED -> PQ_PIPELINE_ABORTED
PQbatchStatus -> PQpipelineStatus (function)
PQenterBatchMode -> PQenterPipelineMode
PQexitBatchMode -> PQexitPipelineMode
PQbatchSendQueue -> PQsendPipeline
PGRES_BATCH_END -> PGRES_PIPELINE_END
PGRES_BATCH_ABORTED -> PGRES_PIPELINE_ABORTED
Also, PQbatchStatus(conn) returns enum PGpipelineStatus (it previously
returned int).
I'm tempted to rename PGASYNC_QUEUED to PGASYNC_PIPELINE_IDLE (not sure
if PGASYNC_PIPELINE_READY fits better with the existing one).
In pgbench, I changed the metacommands to be \startpipeline and
\endpipeline. There's a failing Assert() there which I commented out;
needs fixed.
--
Álvaro Herrera 39°49'30"S 73°17'W
Attachments:
v27-libpq-pipeline.patch (text/x-diff, 95.4K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index b7a82453f0..9f98257dbe 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3099,6 +3099,31 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-end">
+ <term><literal>PGRES_PIPELINE_END</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents the end of a pipeline.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that's
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and it will return this status code,
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_END</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4857,6 +4882,494 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines the application must switch a connection into pipeline mode.
+ Enter pipeline mode with <xref linkend="libpq-PQenterPipelineMode"/>
+ or test whether pipeline mode is active with
+ <xref linkend="libpq-PQpipelineStatus"/>.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using any synchronous command execution functions,
+ such as <function>PQfn</function>, or <function>PQexec</function>
+ and its sibling functions, is an error condition.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill its output buffer and the server's receive
+ buffer before switching to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode the application dispatches requests using
+ normal asynchronous <application>libpq</application> functions, such as:
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQsendDescribePortal</function>,
+ and <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a
+ <xref linkend="libpq-PQsendPipeline"/>
+ call to mark the end of the pipeline. The client need not
+ call <function>PQgetResult</function> immediately after
+ dispatching each operation.
+ <link linkend="libpq-pipeline-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error the server aborts the current
+ transaction and skips processing the rest of the pipeline.
+ Query processing resumes after the end of the failed pipeline.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ pipeline uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one pipeline query, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_END</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_END</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_END</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> call at the corresponding point in
+ the result stream.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ result stream result for the first error and all subsequent results
+ except <literal>PGRES_PIPELINE_END</literal> and null;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries and the expected results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a pipeline causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-pipeline message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the pipeline is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_END</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ still to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/test_libpq/pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-end">
+ <title>Ending Pipeline Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term>
+ <function>PQpipelineStatus</function>
+ <indexterm>
+ <primary>PQpipelineStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current pipeline mode status of the <application>libpq</application>
+ connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error has occurred while processing the current
+ pipeline.
+ The aborted flag is cleared as soon as the result
+ of the <function>PQsendPipeline</function> at the end of the aborted
+ pipeline is processed. Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term>
+ <function>PQenterPipelineMode</function>
+ <indexterm>
+ <primary>PQenterPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term>
+ <function>PQexitPipelineMode</function>
+ <indexterm>
+ <primary>PQexitPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term>
+ <function>PQsendPipeline</function>
+ <indexterm>
+ <primary>PQsendPipeline</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ is failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid sequence. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ must introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in PostgreSQL 14, but clients using
+ the PostgreSQL 14 version of <application>libpq</application> can use
+ pipelines on server versions 7.4 and newer. Pipeline mode works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4897,6 +5410,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index faa7c26b0a..dc3f817879 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1096,6 +1096,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1256,6 +1262,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index a4a3f40048..df75a471ee 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -485,7 +486,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2505,6 +2508,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2694,11 +2701,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2743,8 +2764,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2812,6 +2835,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_END:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3070,13 +3099,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3216,7 +3268,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3460,6 +3520,46 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ /* XXX shouldn't we do PQexitPipelineMode here? */
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4687,7 +4787,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6812,4 +6914,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 8ca0583aa9..6c468c5301 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -536,6 +536,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -567,6 +584,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -6656,6 +6679,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index e730753387..c467bb903b 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_END",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1160,7 +1167,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1186,6 +1193,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in pipeline mode, use PQsendQueryParams\n"));
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, true);
}
@@ -1228,10 +1242,11 @@ PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
conn->last_query = strdup(query);
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
{
/* error message should be set up already */
return 0;
@@ -1296,6 +1311,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1327,6 +1344,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', false, conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1353,32 +1379,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1426,7 +1466,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1447,20 +1488,61 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ case PGASYNC_IDLE:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("internal error, idle state in pipeline mode\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1484,6 +1566,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1493,9 +1579,26 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1605,35 +1708,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1734,14 +1845,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_END.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1810,9 +1924,38 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * In pipeline mode, we prepare the processing of the results
+ * of the next query.
+ */
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * In pipeline mode, query execution state cannot be IDLE as
+ * there can be other queries or results waiting in the queue
+ *
+ * The connection isn't idle since we can't submit new
+ * non-pipeline commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -1993,6 +2136,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2177,6 +2327,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2192,6 +2345,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2200,32 +2365,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2624,6 +2797,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2644,6 +2824,371 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("command queue not clean\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error, IDLE in pipeline mode\n");
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result =
+ PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_END is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline when idle\n");
+ return 0;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3248,6 +3793,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return (pqFlush(conn));
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 6efa53d8b7..b46c6421f4 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -406,6 +406,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in pipeline mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index e4ee9d69d2..8cc619192b 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -217,10 +217,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_END);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -875,6 +893,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index effe0ccf85..6fbb4fe82d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -97,7 +97,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_END, /* end of a pipeline */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -137,6 +140,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -328,6 +341,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -435,6 +449,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 4db498369c..3838697fd5 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,12 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
+ PGASYNC_QUEUED, /* query done, more in current pipeline */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the pqSetenv state machine */
@@ -301,6 +307,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -394,6 +416,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -406,6 +429,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -802,6 +835,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..6e31458be2 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..4fbf97a5b0
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/pipeline
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..64a3a4af5e
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/test_libpq/Makefile
+
+PROGRAM = pipeline
+OBJS = pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/pipeline.c b/src/test/modules/test_libpq/pipeline.c
new file mode 100644
index 0000000000..5e97285e2b
--- /dev/null
+++ b/src/test/modules/test_libpq/pipeline.c
@@ -0,0 +1,969 @@
+/*
+ * src/test/modules/test_libpq/pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_END expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_END)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_END, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_END)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_END)
+ pg_fatal("Unexpected result code %s from second end pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_END.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_END)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_END, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* XXX why do we have a NULL result after PGRES_PIPELINE_END? */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_END)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_END;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_END)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..211629db2d
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,30 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 49614106dc..d1c019c283 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -50,7 +50,8 @@ my @contrib_excludes = (
'pgcrypto', 'sepgsql',
'brin', 'test_extensions',
'test_misc', 'test_pg_dump',
- 'snapshot_too_old', 'unsafe_tests');
+ 'snapshot_too_old', 'unsafe_tests',
+ 'test_libpq');
# Set of variables for frontend modules
my $frontend_defines = { 'initdb' => 'FRONTEND' };
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index bab4f3adb3..c433002c77 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2021-02-16 23:45 ` Zhihong Yu <zyu@yugabyte.com>
2021-02-19 18:35 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2 siblings, 1 reply; 182+ messages in thread
From: Zhihong Yu @ 2021-02-16 23:45 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
Hi,
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode
with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ return CSTATE_ABORTED;
I wonder why the st->state is only assigned for this if block. The state is
not set for other cases where CSTATE_ABORTED is returned.
Should PQ_PIPELINE_OFF be returned for the following case ?
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return false;
Cheers
On Tue, Feb 16, 2021 at 3:14 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:
> Here's a new version, where I've renamed everything to "pipeline". I
> think the docs could use some additional tweaks now in order to make a
> coherent story on pipeline mode, how it can be used in a batched
> fashion, etc.
>
> Here's the renames I applied. It's mostly mechanical, except
> PQbatchSendQueue is now PQsendPipeline:
>
> PQBatchStatus -> PGpipelineStatus (enum)
> PQBATCH_MODE_OFF -> PQ_PIPELINE_OFF
> PQBATCH_MODE_ON -> PQ_PIPELINE_ON
> PQBATCH_MODE_ABORTED -> PQ_PIPELINE_ABORTED
> PQbatchStatus -> PQpipelineStatus (function)
> PQenterBatchMode -> PQenterPipelineMode
> PQexitBatchMode -> PQexitPipelineMode
> PQbatchSendQueue -> PQsendPipeline
> PGRES_BATCH_END -> PGRES_PIPELINE_END
> PGRES_BATCH_ABORTED -> PGRES_PIPELINE_ABORTED
>
> Also, PQbatchStatus(conn) returns enum PGpipelineStatus (it previously
> returned int).
>
> I'm tempted to rename PGASYNC_QUEUED to PGASYNC_PIPELINE_IDLE (not sure
> if PGASYNC_PIPELINE_READY fits better with the existing one).
>
>
> In pgbench, I changed the metacommands to be \startpipeline and
> \endpipeline. There's a failing Assert() there which I commented out;
> needs fixed.
>
> --
> Álvaro Herrera 39°49'30"S 73°17'W
>
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:45 ` Re: PATCH: Batch/pipelining support for libpq Zhihong Yu <zyu@yugabyte.com>
@ 2021-02-19 18:35 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 0 replies; 182+ messages in thread
From: Alvaro Herrera @ 2021-02-19 18:35 UTC (permalink / raw)
To: Zhihong Yu <zyu@yugabyte.com>; +Cc: Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
Hello, thanks for looking at this patch.
On 2021-Feb-16, Zhihong Yu wrote:
> + if (querymode == QUERY_SIMPLE)
> + {
> + commandFailed(st, "startpipeline", "cannot use pipeline mode
> with the simple query protocol");
> + st->state = CSTATE_ABORTED;
> + return CSTATE_ABORTED;
>
> I wonder why the st->state is only assigned for this if block. The state is
> not set for other cases where CSTATE_ABORTED is returned.
Yeah, that's a simple oversight. We don't need to set st->state,
because the caller sets it to the value we return.
> Should PQ_PIPELINE_OFF be returned for the following case ?
>
> +PQpipelineStatus(const PGconn *conn)
> +{
> + if (!conn)
> + return false;
Yep.
Thanks
--
Álvaro Herrera 39°49'30"S 73°17'W
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2021-02-20 00:23 ` Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-20 01:36 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2 siblings, 1 reply; 182+ messages in thread
From: Craig Ringer @ 2021-02-20 00:23 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
On Wed, 17 Feb 2021, 07:13 Alvaro Herrera, <alvherre@alvh.no-ip.org> wrote:
> Here's a new version, where I've renamed everything to "pipeline".
Wow. Thanks.
I
> think the docs could use some additional tweaks now in order to make a
> coherent story on pipeline mode, how it can be used in a batched
> fashion, etc.
>
I'll do that soon and send an update.
>
>
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-20 00:23 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
@ 2021-02-20 01:36 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 0 replies; 182+ messages in thread
From: Alvaro Herrera @ 2021-02-20 01:36 UTC (permalink / raw)
To: Craig Ringer <craig.ringer@enterprisedb.com>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
On 2021-Feb-20, Craig Ringer wrote:
> On Wed, 17 Feb 2021, 07:13 Alvaro Herrera, <alvherre@alvh.no-ip.org> wrote:
> > I
> > think the docs could use some additional tweaks now in order to make a
> > coherent story on pipeline mode, how it can be used in a batched
> > fashion, etc.
>
> I'll do that soon and send an update.
I started to do that, but was sidetracked having a look at error
handling while checking one of the claims.
So now it starts with this:
<sect3 id="libpq-pipeline-sending">
<title>Issuing Queries</title>
<para>
After entering pipeline mode, the application dispatches requests using
<xref linkend="libpq-PQsendQueryParams"/>,
or its prepared-query sibling
<xref linkend="libpq-PQsendQueryPrepared"/>.
These requests are queued on the client-side until flushed to the server;
this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
establish a synchronization point in the pipeline,
or when <xref linkend="libpq-PQflush"/> is called.
The functions <xref linkend="libpq-PQsendPrepare"/>,
<xref linkend="libpq-PQsendDescribePrepared"/>, and
<xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
Result processing is described below.
</para>
<para>
The server executes statements, and returns results, in the order the
client sends them. The server will begin executing the commands in the
pipeline immediately, not waiting for the end of the pipeline.
If any statement encounters an error, the server aborts the current
transaction and skips processing commands in the pipeline until the
next synchronization point established by <function>PQsendPipeline</function>.
(This remains true even if the commands in the pipeline would rollback
the transaction.)
Query processing resumes after the synchronization point.
</para>
(Note I changed the wording that "the pipeline is ended by
PQsendPipeline" to "a synchronization point is established". Is this
easily understandable? On re-reading it, I'm not sure it's really an
improvement.)
BTW we don't explain why doesn't PQsendQuery work (to wit: because it
uses "simple" query protocol and thus doesn't require the Sync message).
I think we should either document that, or change things so that
PQsendQuery no longer uses a 'Q' message when in pipeline mode; instead
use extended query protocol. I don't see why we'd force people to use
PQsendQueryParams when not necessary.
BTW, in my experimentation, the sequence of PGresult that you get when
handling a pipeline with errors don't make a lot of sense. I'll spend
some more time on it.
While at it, as for the PGresult sequence and NULL returns: I think for
PQexec and maybe PQsendQuery, it makes sense to loop until PQgetResult
returns NULL, because you never know how many results are you going to
get. But for PQsendQueryParams, this is no longer true, because you
can't send multiple queries in one query string. So the only way to get
multiple results, is by using single-row mode. But that already has its
own protocol for multiple results, namely to get a stream of
PGRES_SINGLE_TUPLE terminated by a zero-rows PGRES_TUPLES_OK. So I'm
not sure there is a strong need for the mandatory NULL result.
--
Álvaro Herrera 39°49'30"S 73°17'W
^ permalink raw reply [nested|flat] 182+ messages in thread
* RE: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2021-03-03 07:07 ` k.jamison@fujitsu.com <k.jamison@fujitsu.com>
2021-03-03 16:34 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2 siblings, 1 reply; 182+ messages in thread
From: k.jamison@fujitsu.com @ 2021-03-03 07:07 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; Craig Ringer <craig.ringer@enterprisedb.com>; +Cc: Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
On Wed, Feb 17, 2021 8:14 AM (JST), Alvaro Herrera wrote:
Hi Alvaro,
> Here's a new version, where I've renamed everything to "pipeline". I think the
> docs could use some additional tweaks now in order to make a coherent
> story on pipeline mode, how it can be used in a batched fashion, etc.
I tried applying this patch to test it on top of Iwata-san's libpq trace log [1].
In my environment, the compiler complains.
It seems that in libpqwalreceiver.c: libpqrcv_exec()
the switch for PQresultStatus needs to handle the
cases for PGRES_PIPELINE_END and PGRES_PIPELINE_ABORTED too.
switch (PQresultStatus(pgres))
{
...
Case PGRES_PIPELINE_ABORTED:
...
Case PGRES_PIPELINE_ END:
Regards,
Kirk Jamison
[1] https://www.postgresql.org/message-id/flat/1d650278-552a-4bd2-8411-a907fd54446c%40www.fastmail.com#3...
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-03-03 07:07 ` RE: PATCH: Batch/pipelining support for libpq k.jamison@fujitsu.com <k.jamison@fujitsu.com>
@ 2021-03-03 16:34 ` 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-03 23:42 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-04 00:45 ` Re: PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 2 replies; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-03 16:34 UTC (permalink / raw)
To: k.jamison@fujitsu.com <k.jamison@fujitsu.com>; +Cc: Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
On 2021-Mar-03, k.jamison@fujitsu.com wrote:
> I tried applying this patch to test it on top of Iwata-san's libpq trace log [1].
> In my environment, the compiler complains.
> It seems that in libpqwalreceiver.c: libpqrcv_exec()
> the switch for PQresultStatus needs to handle the
> cases for PGRES_PIPELINE_END and PGRES_PIPELINE_ABORTED too.
Yeah, I noticed this too and decided to handle these cases as errors.
Here's v28, which includes that fix, and also gets rid of the
PGASYNC_QUEUED state; I instead made the rest of the code work using the
existing PGASYNC_IDLE state. This allows us get rid of a few cases
where we had to report "internal error: IDLE state unexpected" in a few
cases, and was rather pointless. With the current formulation, both
pipeline mode and normal mode share pretty much all state.
Also, I've renamed PGRES_PIPELINE_END to PGRES_PIPELINE_SYNC, because
that's closer to what it means: that result state by no means that
pipeline mode has ended. It only means that you called PQsendPipeline()
so the server gives you a Sync message.
I'm much more comfortable with this version, so I'm marking the patch as
Ready for Committer in case anybody else wants to look at this before I
push it.
However: on errors, I think it's a bit odd that you get a NULL from
PQgetResult after getting PGRES_PIPELINE_ABORTED. Maybe I should
suppress that. I'm no longer wrestling with the NULL after
PGRES_PIPELINE_END however, because that's not critical and you can not
ask for it and things work fine.
(Also, the test pipeline.c program in src/test/modules/test_libpq has a
new "transaction" mode that is not exercised by the TAP program; I'll
plug that in before commit.)
--
Álvaro Herrera Valdivia, Chile
Attachments:
v28-libpq-pipeline.patch (text/x-diff, 100.9K) download
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index cc20b0f234..4ea8cb8c93 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3199,6 +3199,32 @@ ExecStatusType PQresultStatus(const PGresult *res);
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-sync">
+ <term><literal>PGRES_PIPELINE_SYNC</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a point in a
+ pipeline where a synchronization point has been established.
+ This status occurs only when pipeline mode has been selected.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-pgres-pipeline-aborted">
+ <term><literal>PGRES_PIPELINE_ABORTED</literal></term>
+ <listitem>
+ <para>
+ The <structname>PGresult</structname> represents a pipeline that's
+ received an error from the server. <function>PQgetResult</function>
+ must be called repeatedly, and it will return this status code,
+ until the end of the current pipeline, at which point it will return
+ <literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
+ resume.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
If the result status is <literal>PGRES_TUPLES_OK</literal> or
@@ -4957,6 +4983,498 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-pipeline-mode">
+ <title>Pipeline Mode</title>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>libpq</primary>
+ <secondary>pipeline mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>pipelining</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-pipeline-mode">
+ <primary>batch mode</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> pipeline mode allows applications to
+ send a query without having to read the result of the previously
+ sent query. Taking advantage of the pipeline mode, a client will wait
+ less for the server, since multiple queries/results can be sent/
+ received in a single network transaction.
+ </para>
+
+ <para>
+ While pipeline mode provides a significant performance boost, writing
+ clients using the pipeline mode is more complex because it involves
+ managing a queue of pending queries and finding which result
+ corresponds to which query in the queue.
+ </para>
+
+ <para>
+ Pipeline mode also generally consumes more memory on both the client and server,
+ though careful and aggressive management of the send/receive queue can mitigate
+ this. This applies whether or not the connection is in blocking or non-blocking
+ mode.
+ </para>
+
+ <sect2 id="libpq-pipeline-using">
+ <title>Using Pipeline Mode</title>
+
+ <para>
+ To issue pipelines the application must switch a connection into pipeline mode.
+ Enter pipeline mode with <xref linkend="libpq-PQenterPipelineMode"/>
+ or test whether pipeline mode is active with
+ <xref linkend="libpq-PQpipelineStatus"/>.
+ In pipeline mode, only <link linkend="libpq-async">asynchronous operations</link>
+ are permitted, and <literal>COPY</literal> is disallowed.
+ Using any synchronous command execution functions,
+ such as <function>PQfn</function>, or <function>PQexec</function>
+ and its sibling functions, is an error condition.
+ </para>
+
+ <note>
+ <para>
+ It is best to use pipeline mode with <application>libpq</application> in
+ <link linkend="libpq-PQsetnonblocking">non-blocking mode</link>. If used
+ in blocking mode it is possible for a client/server deadlock to occur.
+ <footnote>
+ <para>
+ The client will block trying to send queries to the server, but the
+ server will block trying to send results to the client from queries
+ it has already processed. This only occurs when the client sends
+ enough queries to fill both its output buffer and the server's receive
+ buffer before switching to processing input from the server,
+ but it's hard to predict exactly when that will happen.
+ </para>
+ </footnote>
+ </para>
+ </note>
+
+ <sect3 id="libpq-pipeline-sending">
+ <title>Issuing Queries</title>
+
+ <para>
+ After entering pipeline mode, the application dispatches requests using
+ <xref linkend="libpq-PQsendQueryParams"/>,
+ or its prepared-query sibling
+ <xref linkend="libpq-PQsendQueryPrepared"/>.
+ These requests are queued on the client-side until flushed to the server;
+ this occurs when <xref linkend="libpq-PQsendPipeline"/> is used to
+ establish a synchronization point in the pipeline,
+ or when <xref linkend="libpq-PQflush"/> is called.
+ The functions <xref linkend="libpq-PQsendPrepare"/>,
+ <xref linkend="libpq-PQsendDescribePrepared"/>, and
+ <xref linkend="libpq-PQsendDescribePortal"/> also work in pipeline mode.
+ Result processing is described below.
+ </para>
+
+ <para>
+ The server executes statements, and returns results, in the order the
+ client sends them. The server will begin executing the commands in the
+ pipeline immediately, not waiting for the end of the pipeline.
+ If any statement encounters an error, the server aborts the current
+ transaction and skips processing commands in the pipeline until the
+ next synchronization point established by <function>PQsendPipeline</function>.
+ (This remains true even if the commands in the pipeline would rollback
+ the transaction.)
+ Query processing resumes after the synchronization point.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one; for example, one query may define a table that the next
+ query in the same pipeline uses. Similarly, an application may
+ create a named prepared statement and execute it with later
+ statements in the same pipeline.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-results">
+ <title>Processing Results</title>
+
+ <para>
+ To process the result of one pipeline query, the application calls
+ <function>PQgetResult</function> repeatedly and handles each result
+ until <function>PQgetResult</function> returns null.
+ The result from the next query in the pipeline may then be retrieved using
+ <function>PQgetResult</function> again and the cycle repeated.
+ The application handles individual statement results as normal.
+ When the results of all the queries in the pipeline have been
+ returned, <function>PQgetResult</function> returns a result
+ containing the status value <literal>PGRES_PIPELINE_SYNC</literal>.
+ </para>
+
+ <para>
+ The client may choose to defer result processing until the complete
+ pipeline has been sent, or interleave that with sending further
+ queries in the pipeline; see <xref linkend="libpq-pipeline-interleave" />.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function>
+ before retrieving results with <function>PQgetResult</function>.
+ This mode selection is effective only for the query currently
+ being processed. For more information on the use of
+ <function>PQsetSingleRowMode</function>,
+ refer to <xref linkend="libpq-single-row-mode"/>.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal
+ asynchronous processing except that it may contain the new
+ <type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
+ and <literal>PGRES_PIPELINE_ABORTED</literal>.
+ <literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
+ <function>PQsendPipeline</function> call at the corresponding point in
+ the result stream.
+ <literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
+ result stream result for the first error and all subsequent results
+ except <literal>PGRES_PIPELINE_SYNC</literal> and null;
+ see <xref linkend="libpq-pipeline-errors"/>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing pipeline results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed (except that
+ <function>PQgetResult</function> returns null to indicate that we start
+ returning the results of next query). The application must keep track
+ of the order in which it sent queries and the expected results.
+ Applications will typically use a state machine or a FIFO queue for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-pipeline-errors">
+ <title>Error Handling</title>
+
+ <para>
+ When a query in a pipeline causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the pipeline
+ synchronization message. The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the pipeline is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
+ operation in an aborted pipeline. The result for
+ <function>PQsendPipeline</function> is reported as
+ <literal>PGRES_PIPELINE_SYNC</literal> to signal the end of the aborted pipeline
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the pipeline used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ pipeline starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the pipeline. If a pipeline contains
+ <emphasis>multiple explicit transactions</emphasis>, all transactions that
+ committed prior to the error remain committed, the currently in-progress
+ transaction is aborted, and all subsequent operations are skipped completely,
+ including subsequent transactions.
+ </para>
+
+ <note>
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal> — only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously, the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+ </note>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-interleave">
+ <title>Interleaving Result Processing and Query Dispatch</title>
+
+ <para>
+ To avoid deadlocks on large pipelines the client should be structured
+ around a non-blocking event loop using operating system facilities
+ such as <function>select</function>, <function>poll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work
+ still to be dispatched and a queue of work that has been dispatched
+ but not yet had its results processed. When the socket is writable
+ it should dispatch more work. When the socket is readable it should
+ read results and process them, matching them up to the next entry in
+ its expected results queue. Based on available memory, results from
+ socket should be read frequently: there's no need to wait until the
+ pipeline end to read the results. Pipelines should be scoped to logical
+ units of work, usually (but not necessarily) one transaction per pipeline.
+ There's no need to exit pipeline mode and re-enter it between pipelines,
+ or to wait for one pipeline to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state
+ machine to track sent and received work is in
+ <filename>src/test/modules/test_libpq/pipeline.c</filename>
+ in the PostgreSQL source distribution.
+ </para>
+ </sect3>
+
+ <sect3 id="libpq-pipeline-end">
+ <title>Ending Pipeline Mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed, and
+ the end pipeline result has been consumed, the application may return
+ to non-pipelined mode with <xref linkend="libpq-PQexitPipelineMode"/>.
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-functions">
+ <title>Functions Associated with Pipeline Mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQpipelineStatus">
+ <term>
+ <function>PQpipelineStatus</function>
+ <indexterm>
+ <primary>PQpipelineStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current pipeline mode status of the <application>libpq</application>
+ connection.
+<synopsis>
+PGpipelineStatus PQpipelineStatus(const PGconn *conn);
+</synopsis>
+ </para>
+
+ <para>
+ <function>PQpipelineStatus</function> can return one of the following values:
+
+ <variablelist>
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ON</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in
+ pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_OFF</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is
+ <emphasis>not</emphasis> in pipeline mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>
+ <literal>PQ_PIPELINE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ The <application>libpq</application> connection is in pipeline
+ mode and an error has occurred while processing the current
+ pipeline.
+ The aborted flag is cleared as soon as the result
+ of the <function>PQsendPipeline</function> at the end of the aborted
+ pipeline is processed. Clients don't usually need this function to
+ verify aborted status, as they can tell that the pipeline is aborted
+ from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQenterPipelineMode">
+ <term>
+ <function>PQenterPipelineMode</function>
+ <indexterm>
+ <primary>PQenterPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter pipeline mode if it is currently idle or
+ already in pipeline mode.
+
+<synopsis>
+int PQenterPipelineMode(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success.
+ Returns 0 and has no effect if the connection is not currently
+ idle, i.e., it has a result ready, or it is waiting for more
+ input from the server, etc.
+ This function does not actually send anything to the server,
+ it just changes the <application>libpq</application> connection
+ state.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQexitPipelineMode">
+ <term>
+ <function>PQexitPipelineMode</function>
+ <indexterm>
+ <primary>PQexitPipelineMode</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit pipeline mode if it is currently in pipeline mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQexitPipelineMode(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 1 and takes no action if not in
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQsendPipeline">
+ <term>
+ <function>PQsendPipeline</function>
+ <indexterm>
+ <primary>PQsendPipeline</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Marks a synchronization point in a pipeline by sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ and flushing the send buffer. This serves as
+ the delimiter of an implicit transaction and an error recovery
+ point; see <xref linkend="libpq-pipeline-errors"/>.
+
+<synopsis>
+int PQsendPipeline(PGconn *conn);
+</synopsis>
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 if the connection is not in
+ pipeline mode or sending a
+ <link linkend="protocol-flow-ext-query">sync message</link>
+ is failed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect2>
+
+ <sect2 id="libpq-pipeline-tips">
+ <title>When to Use Pipeline Mode</title>
+
+ <para>
+ Much like asynchronous query mode, there is no meaningful performance
+ overhead when using pipeline mode. It increases client application complexity,
+ and extra caution is required to prevent client/server deadlocks, but
+ pipeline mode can offer considerable performance improvements, in exchange for
+ increased memory usage from leaving state around longer.
+ </para>
+
+ <para>
+ Pipeline mode is most useful when the server is distant, i.e., network latency
+ (<quote>ping time</quote>) is high, and also when many small operations
+ are being performed in rapid sequence. There is usually less benefit
+ in using pipelined commands when each query takes many multiples of the client/server
+ round-trip time to execute. A 100-statement operation run on a server
+ 300ms round-trip-time away would take 30 seconds in network latency alone
+ without pipelining; with pipelining it may spend as little as 0.3s waiting for
+ results from the server.
+ </para>
+
+ <para>
+ Use pipelined commands when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed
+ into operations on sets, or into a <literal>COPY</literal> operation.
+ </para>
+
+ <para>
+ Pipeline mode is not useful when information from one operation is required by
+ the client to produce the next operation. In such cases, the client
+ must introduce a synchronization point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+BEGIN;
+SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+-- result: x=2
+-- client adds 1 to x:
+UPDATE mytable SET x = 3 WHERE id = 42;
+COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <para>
+ Pipelining is less useful, and more complex, when a single pipeline contains
+ multiple transactions (see <xref linkend="libpq-pipeline-errors"/>).
+ </para>
+
+ <note>
+ <para>
+ The pipeline API was introduced in PostgreSQL 14, but clients using
+ the PostgreSQL 14 version of <application>libpq</application> can use
+ pipelines on server versions 7.4 and newer. Pipeline mode works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+ </sect2>
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-by-Row</title>
@@ -4997,6 +5515,13 @@ int PQflush(PGconn *conn);
Each object should be freed with <xref linkend="libpq-PQclear"/> as usual.
</para>
+ <para>
+ When using pipeline mode, single-row mode needs to be activated for each
+ query in the pipeline before retrieving results for that query
+ with <function>PQgetResult</function>.
+ See <xref linkend="libpq-pipeline-mode"/> for more information.
+ </para>
+
<para>
<variablelist>
<varlistentry id="libpq-PQsetSingleRowMode">
diff --git a/doc/src/sgml/lobj.sgml b/doc/src/sgml/lobj.sgml
index 6d46da42e2..012e44c736 100644
--- a/doc/src/sgml/lobj.sgml
+++ b/doc/src/sgml/lobj.sgml
@@ -130,6 +130,10 @@
<application>libpq</application> library.
</para>
+ <para>
+ Client applications cannot use these functions while a libpq connection is in pipeline mode.
+ </para>
+
<sect2 id="lo-create">
<title>Creating a Large Object</title>
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2ec0580a79..75448162ca 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -1109,6 +1109,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
row, the last value is kept.
</para>
+ <para>
+ <literal>\gset</literal> and <literal>\aset</literal> cannot be used
+ pipeline mode, since query results are not immediately
+ fetched in this mode.
+ </para>
+
<para>
The following example puts the final account balance from the first query
into variable <replaceable>abalance</replaceable>, and fills variables
@@ -1269,6 +1275,21 @@ SELECT 4 AS four \; SELECT 5 AS five \aset
</programlisting></para>
</listitem>
</varlistentry>
+
+ <varlistentry id='pgbench-metacommand-pipeline'>
+ <term><literal>\startpipeline</literal></term>
+ <term><literal>\endpipeline</literal></term>
+
+ <listitem>
+ <para>
+ These commands delimit the start and end of a pipeline of SQL statements.
+ In a pipeline, statements are sent to server without waiting for the results
+ of previous statements (see <xref linkend="libpq-pipeline-mode"/>).
+ Pipeline mode requires the extended query protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect2>
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 5272eed9ab..f74378110a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -1019,6 +1019,12 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
walres->err = _("empty query");
break;
+ case PGRES_PIPELINE_SYNC:
+ case PGRES_PIPELINE_ABORTED:
+ walres->status = WALRCV_ERROR;
+ walres->err = _("unexpected pipeline mode");
+ break;
+
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
case PGRES_BAD_RESPONSE:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31a4df45f5..fbbe270654 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -351,10 +351,11 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
- * and we enter the CSTATE_SLEEP state to wait for it to expire. Other
- * meta-commands are executed immediately. If the command about to start
- * is actually beyond the end of the script, advance to CSTATE_END_TX.
+ * CSTATE_WAIT_RESULT state unless in pipeline mode. On a \sleep
+ * meta-command, the timer is set, and we enter the CSTATE_SLEEP state to
+ * wait for it to expire. Other meta-commands are executed immediately. If
+ * the command about to start is actually beyond the end of the script,
+ * advance to CSTATE_END_TX.
*
* CSTATE_WAIT_RESULT waits until we get a result set back from the server
* for the current command.
@@ -484,7 +485,9 @@ typedef enum MetaCommand
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
- META_ENDIF /* \endif */
+ META_ENDIF, /* \endif */
+ META_STARTPIPELINE, /* \startpipeline */
+ META_ENDPIPELINE /* \endpipeline */
} MetaCommand;
typedef enum QueryMode
@@ -2504,6 +2507,10 @@ getMetaCommand(const char *cmd)
mc = META_GSET;
else if (pg_strcasecmp(cmd, "aset") == 0)
mc = META_ASET;
+ else if (pg_strcasecmp(cmd, "startpipeline") == 0)
+ mc = META_STARTPIPELINE;
+ else if (pg_strcasecmp(cmd, "endpipeline") == 0)
+ mc = META_ENDPIPELINE;
else
mc = META_NONE;
return mc;
@@ -2693,11 +2700,25 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In pipeline mode, we use asynchronous functions. If a
+ * server-side error occurs, it will be processed later
+ * among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ pg_log_error("%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2741,8 +2762,10 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
* varprefix should be set only with \gset or \aset, and SQL commands do
* not need it.
*/
+#if 0
Assert((meta == META_NONE && varprefix == NULL) ||
((meta == META_GSET || meta == META_ASET) && varprefix != NULL));
+#endif
res = PQgetResult(st->con);
@@ -2810,6 +2833,12 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* otherwise the result is simply thrown away by PQclear below */
break;
+ case PGRES_PIPELINE_SYNC:
+ pg_log_debug("client %d pipeline ending", st->id);
+ if (PQexitPipelineMode(st->con) != 1)
+ pg_log_error("client %d failed to exit pipeline mode", st->id);
+ break;
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3066,13 +3095,36 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
/* Execute the command */
if (command->type == SQL_COMMAND)
{
+ /* disallow \aset and \gset in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (command->meta == META_GSET)
+ {
+ commandFailed(st, "gset", "\\gset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (command->meta == META_ASET)
+ {
+ commandFailed(st, "aset", "\\aset is not allowed in pipeline mode");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
if (!sendCommand(st, command))
{
commandFailed(st, "SQL", "SQL command send failed");
st->state = CSTATE_ABORTED;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in pipeline mode */
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -3212,7 +3264,15 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (readCommandResponse(st,
sql_script[st->use_file].commands[st->command]->meta,
sql_script[st->use_file].commands[st->command]->varprefix))
- st->state = CSTATE_END_COMMAND;
+ {
+ /*
+ * outside of pipeline mode: stop reading results.
+ * pipeline mode: continue reading results until an
+ * end-of-pipeline response.
+ */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ st->state = CSTATE_END_COMMAND;
+ }
else
st->state = CSTATE_ABORTED;
break;
@@ -3456,6 +3516,49 @@ executeMetaCommand(CState *st, instr_time *now)
return CSTATE_ABORTED;
}
}
+ else if (command->meta == META_STARTPIPELINE)
+ {
+ /*
+ * In pipeline mode, we use a workflow based on libpq pipeline
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "startpipeline", "cannot use pipeline mode with the simple query protocol");
+ return CSTATE_ABORTED;
+ }
+
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ commandFailed(st, "startpipeline", "already in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (PQenterPipelineMode(st->con) == 0)
+ {
+ commandFailed(st, "startpipeline", "failed to enter pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ }
+ else if (command->meta == META_ENDPIPELINE)
+ {
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
+ {
+ commandFailed(st, "endpipeline", "not in pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ if (!PQsendPipeline(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to send the pipeline");
+ return CSTATE_ABORTED;
+ }
+ if (!PQexitPipelineMode(st->con))
+ {
+ commandFailed(st, "endpipeline", "failed to exit pipeline mode");
+ return CSTATE_ABORTED;
+ }
+ /* collect pending results before getting out of pipeline mode */
+ return CSTATE_WAIT_RESULT;
+ }
/*
* executing the expression or shell command might have taken a
@@ -4683,7 +4786,9 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
- else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
+ else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF ||
+ my_command->meta == META_STARTPIPELINE ||
+ my_command->meta == META_ENDPIPELINE)
{
if (my_command->argc != 1)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
@@ -6808,4 +6913,5 @@ pthread_join(pthread_t th, void **thread_return)
return 0;
}
+
#endif /* WIN32 */
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index bbc1f90481..60d09e6d63 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -179,3 +179,7 @@ PQgetgssctx 176
PQsetSSLKeyPassHook_OpenSSL 177
PQgetSSLKeyPassHook_OpenSSL 178
PQdefaultSSLKeyPassHook_OpenSSL 179
+PQenterPipelineMode 180
+PQexitPipelineMode 181
+PQsendPipeline 182
+PQpipelineStatus 183
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 9812a14662..3fa305f89a 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -540,6 +540,23 @@ pqDropConnection(PGconn *conn, bool flushInput)
}
}
+/*
+ * pqFreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+pqFreeCommandQueue(PGcommandQueueEntry *queue)
+{
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *cur = queue;
+
+ queue = cur->next;
+ if (cur->query)
+ free(cur->query);
+ free(cur);
+ }
+}
/*
* pqDropServerData
@@ -571,6 +588,12 @@ pqDropServerData(PGconn *conn)
}
conn->notifyHead = conn->notifyTail = NULL;
+ pqFreeCommandQueue(conn->cmd_queue_head);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ pqFreeCommandQueue(conn->cmd_queue_recycle);
+ conn->cmd_queue_recycle = NULL;
+
/* Reset ParameterStatus data, as well as variables deduced from it */
pstatus = conn->pstatus;
while (pstatus != NULL)
@@ -2490,6 +2513,7 @@ keep_going: /* We will come back to here until there is
/* Drop any PGresult we might have, too */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn);
/* Reset conn->status to put the state machine in the right state */
@@ -4039,6 +4063,7 @@ makeEmptyPGconn(void)
conn->status = CONNECTION_BAD;
conn->asyncStatus = PGASYNC_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
conn->xactStatus = PQTRANS_IDLE;
conn->options_valid = false;
conn->nonblocking = false;
@@ -4299,6 +4324,7 @@ closePGconn(PGconn *conn)
conn->status = CONNECTION_BAD; /* Well, not really _bad_ - just absent */
conn->asyncStatus = PGASYNC_IDLE;
conn->xactStatus = PQTRANS_IDLE;
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
pqClearAsyncResult(conn); /* deallocate result */
resetPQExpBuffer(&conn->errorMessage);
release_conn_addrinfo(conn);
@@ -6850,6 +6876,15 @@ PQbackendPID(const PGconn *conn)
return conn->be_pid;
}
+PGpipelineStatus
+PQpipelineStatus(const PGconn *conn)
+{
+ if (!conn)
+ return PQ_PIPELINE_OFF;
+
+ return conn->pipelineStatus;
+}
+
int
PQconnectionNeedsPassword(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index a550753855..e386ea92f4 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_PIPELINE_SYNC",
+ "PGRES_PIPELINE_ABORTED"
};
/*
@@ -71,6 +73,11 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *pqMakePipelineCmd(PGconn *conn);
+static void pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry);
+static void pqPipelineProcessQueue(PGconn *conn);
+static int pqPipelineFlush(PGconn *conn);
/* ----------------
@@ -1171,7 +1178,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1197,18 +1204,37 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQuery");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, true);
}
int
PQsendQueryContinue(PGconn *conn, const char *query)
{
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot use %s in pipeline mode, use PQsendQueryParams\n"),
+ "PQsendQueryContinue");
+ return 0;
+ }
+
return PQsendQueryInternal(conn, query, false);
}
static int
PQsendQueryInternal(PGconn *conn, const char *query, bool newQuery)
{
+ /* pipeline mode requires extended query protocol */
+ Assert(conn->pipelineStatus == PQ_PIPELINE_OFF);
+
if (!PQsendQueryStart(conn, newQuery))
return 0;
@@ -1307,6 +1333,8 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+
if (!PQsendQueryStart(conn, true))
return 0;
@@ -1338,6 +1366,15 @@ PQsendPrepare(PGconn *conn,
return 0;
}
+ /* Alloc pipeline memory before doing anything */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+ }
+
/* construct the Parse message */
if (pqPutMsgStart('P', false, conn) < 0 ||
pqPuts(stmtName, conn) < 0 ||
@@ -1364,32 +1401,46 @@ PQsendPrepare(PGconn *conn,
if (pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /*
+ * In non-pipeline mode, add a Sync and prepare to send. In pipeline mode
+ * we just keep track of the new message.
+ */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ /* remember we are doing just a Parse */
+ conn->queryclass = PGQUERY_PREPARE;
- /* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
-
- /* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ /* and remember the query text too, if possible */
+ /* if insufficient memory, last_query just winds up NULL */
+ if (conn->last_query)
+ free(conn->last_query);
+ conn->last_query = strdup(query);
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ else
+ {
+ pipeCmd->queryclass = PGQUERY_PREPARE;
+ /* as above, if insufficient memory, query winds up NULL */
+ pipeCmd->query = strdup(query);
+ pqAppendPipelineCmd(conn, pipeCmd);
+ }
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
- /* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1437,7 +1488,8 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn, bool newQuery)
@@ -1458,20 +1510,57 @@ PQsendQueryStart(PGconn *conn, bool newQuery)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE &&
+ conn->pipelineStatus == PQ_PIPELINE_OFF)
{
appendPQExpBufferStr(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * When enqueuing commands we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when pqPipelineProcessQueue()
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ }
+ }
+ else
+ {
+ /*
+ * This command's results will come in immediately. Initialize async
+ * result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1495,6 +1584,9 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **query;
+ PGQueryClass *queryclass;
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1505,8 +1597,29 @@ PQsendQueryGuts(PGconn *conn,
}
/*
- * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
- * using specified statement name and the unnamed portal.
+ * Decide where the query is going to be stored. In pipeline mode, we
+ * allocate a new pipeline element; in non-pipeline mode, it's simply the
+ * connection's last query.
+ */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+ /*
+ * We will send Parse (if needed), Bind, Describe Portal, Execute, Sync
+ * (if not in pipeline mode), using specified statement name and the
+ * unnamed portal.
*/
if (command)
@@ -1616,35 +1729,43 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ /* construct the Sync message if not in pipeline mode */
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
- /* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ /* if insufficient memory, query just winds up NULL */
+ if (*query)
+ free(*query);
if (command)
- conn->last_query = strdup(command);
+ *query = strdup(command);
else
- conn->last_query = NULL;
+ *query = NULL;
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -1745,14 +1866,17 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY || conn->write_failed;
}
-
/*
* PQgetResult
* Get the next PGresult produced by a query. Returns NULL if no
* query work remains or an error has occurred (e.g. out of
* memory).
+ *
+ * In pipeline mode, once all the result of a query have been returned,
+ * PQgetResult returns NULL to let the user know that the next
+ * query is being processed. At the end of the pipeline, returns a
+ * result with PQresultStatus(result) == PGRES_PIPELINE_SYNC.
*/
-
PGresult *
PQgetResult(PGconn *conn)
{
@@ -1822,8 +1946,36 @@ PQgetResult(PGconn *conn)
{
case PGASYNC_IDLE:
res = NULL; /* query is complete */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to return the NULL that terminates the round
+ * of results from the current query; prepare to send the
+ * results of the next query when we're called next. Also,
+ * since this is the start of the results of the next query,
+ * clear any prior error message.
+ */
+ resetPQExpBuffer(&conn->errorMessage);
+ pqPipelineProcessQueue(conn);
+ }
break;
case PGASYNC_READY:
+ res = pqPrepareAsyncResult(conn);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ /*
+ * We're about to send the results of the current query. Set
+ * us idle so that we send a terminating NULL next, above.
+ */
+ conn->asyncStatus = PGASYNC_IDLE;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
@@ -2004,6 +2156,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("synchronous command execution functions are not allowed in pipeline mode\n"));
+ return false;
+ }
+
/*
* Since this is the beginning of a query cycle, reset the error buffer.
*/
@@ -2188,6 +2347,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2203,6 +2365,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ pipeCmd = pqMakePipelineCmd(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ queryclass = &conn->queryclass;
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2211,32 +2385,40 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
- /* reset last_query string (not relevant now) */
- if (conn->last_query)
+ /* reset last-query string (not relevant now) */
+ if (conn->last_query && conn->pipelineStatus != PQ_PIPELINE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
}
/*
- * Give the data a push. In nonblock mode, don't complain if we're unable
- * to send it all; PQgetResult() will do any additional flushing needed.
+ * Give the data a push (in pipeline mode, only if we're past the size
+ * threshold). In nonblock mode, don't complain if we're unable to send
+ * it all; PQgetResult() will do any additional flushing needed.
*/
- if (pqFlush(conn) < 0)
+ if (pqPipelineFlush(conn) < 0)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ pqAppendPipelineCmd(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ pqRecyclePipelineCmd(conn, pipeCmd);
/* error message should be set up already */
return 0;
}
@@ -2635,6 +2817,13 @@ PQfn(PGconn *conn,
*/
resetPQExpBuffer(&conn->errorMessage);
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("PQfn not allowed in pipeline mode\n"));
+ return NULL;
+ }
+
if (conn->sock == PGINVALID_SOCKET || conn->asyncStatus != PGASYNC_IDLE ||
conn->result != NULL)
{
@@ -2655,6 +2844,359 @@ PQfn(PGconn *conn,
args, nargs);
}
+/* ====== Pipeline mode support ======== */
+
+/*
+ * PQenterPipelineMode
+ * Put an idle connection in pipeline mode.
+ *
+ * Returns 1 on success. On failure, errorMessage is set and 0 is returned.
+ *
+ * Commands submitted after this can be pipelined on the connection;
+ * there's no requirement to wait for one to finish before the next is
+ * dispatched.
+ *
+ * Queuing of a new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQsendPipeline. Multiple pipelines
+ * can be sent while in pipeline mode. Pipeline mode can be exited
+ * by calling PQexitPipelineMode() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQenterPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ /* succeed with no action if already in pipeline mode */
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ return 1;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot enter pipeline mode, connection not idle\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+
+ return 1;
+}
+
+/*
+ * PQexitPipelineMode
+ * End pipeline mode and return to normal command mode.
+ *
+ * Returns 1 in success (pipeline mode successfully ended, or not in pipeline
+ * mode).
+ *
+ * Returns 0 if in pipeline mode and cannot be ended yet. Error message will
+ * be set.
+ */
+int
+PQexitPipelineMode(PGconn *conn)
+{
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ return 1;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ /* there are some uncollected results */
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+
+ case PGASYNC_BUSY:
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode while busy\n"));
+ return 0;
+
+ default:
+ /* OK */
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot exit pipeline mode with uncollected results\n"));
+ return 0;
+ }
+
+ conn->pipelineStatus = PQ_PIPELINE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ /* Flush any pending data in out buffer */
+ if (pqFlush(conn) < 0)
+ return 0; /* error message is setup already */
+ return 1;
+}
+
+/*
+ * pqPipelineProcessQueue: subroutine for PQgetResult
+ * In pipeline mode, start processing the results of the next query in the queue.
+ */
+static void
+pqPipelineProcessQueue(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ Assert(conn->pipelineStatus != PQ_PIPELINE_OFF);
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: COPY in pipeline mode\n");
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return;
+ case PGASYNC_IDLE:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In pipeline mode but nothing left on the queue; caller can submit
+ * more work or PQexitPipelineMode() now.
+ */
+ return;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-pipeline call.
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /*
+ * Reset single-row processing mode. (Client has to set it up for each
+ * query, if desired.)
+ */
+ conn->singleRowMode = false;
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ pqRecyclePipelineCmd(conn, next_query);
+
+ if (conn->pipelineStatus == PQ_PIPELINE_ABORTED &&
+ conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted pipeline we don't get anything from the server for
+ * each result; we're just discarding input until we get to the next
+ * sync from the server. The client needs to know its queries got
+ * aborted so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn, PGRES_PIPELINE_ABORTED);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ pqSaveErrorResult(conn);
+ return;
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+}
+
+/*
+ * PQsendPipeline
+ * Send a Sync message as part of a pipeline, and flush to server
+ *
+ * It's legal to start submitting more commands in the pipeline immediately,
+ * without waiting for the results of the current pipeline. There's no need to
+ * end pipeline mode and start it again.
+ *
+ * If a command in a pipeline fails, every subsequent command up to and including
+ * the result to the Sync message sent by PQsendPipeline gets set to
+ * PGRES_PIPELINE_ABORTED state. If the whole pipeline is processed without
+ * error, a PGresult with PGRES_PIPELINE_SYNC is produced.
+ *
+ * Queries can already have been sent before PQsendPipeline is called, but
+ * PQsendPipeline need to be called before retrieving command results.
+ *
+ * The connection will remain in pipeline mode and unavailable for new
+ * synchronous command execution functions until all results from the pipeline
+ * are processed by the client.
+ */
+int
+PQsendPipeline(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return 0;
+
+ if (conn->pipelineStatus == PQ_PIPELINE_OFF)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot send pipeline when not in pipeline mode\n"));
+ return 0;
+ }
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ /* should be unreachable */
+ appendPQExpBufferStr(&conn->errorMessage,
+ "internal error: cannot send pipeline while in COPY\n");
+ return 0;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_IDLE:
+ /* OK to send sync */
+ break;
+ }
+
+ entry = pqMakePipelineCmd(conn);
+ if (entry == NULL)
+ return 0; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ pqAppendPipelineCmd(conn, entry);
+
+ /*
+ * Give the data a push. In nonblock mode, don't complain if we're unable
+ * to send it all; PQgetResult() will do any additional flushing needed.
+ */
+ if (PQflush(conn) < 0)
+ goto sendFailed;
+
+ /*
+ * Call pqPipelineProcessQueue so the user can call start calling
+ * PQgetResult.
+ */
+ pqPipelineProcessQueue(conn);
+
+ return 1;
+
+sendFailed:
+ pqRecyclePipelineCmd(conn, entry);
+ /* error message should be set up already */
+ return 0;
+}
+
+/*
+ * pqMakePipelineCmd
+ * Get a command queue entry for caller to fill.
+ *
+ * If the recycle queue has a free element, that is returned; if not, a
+ * fresh one is allocated. Caller is responsible for adding it to the
+ * command queue (pqAppendPipelineCmd) once the struct is filled in, or
+ * releasing the memory (pqRecyclePipelineCmd) if an error occurs.
+ *
+ * If allocation fails, sets the error message and returns NULL.
+ */
+static PGcommandQueueEntry *
+pqMakePipelineCmd(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * pqAppendPipelineCmd
+ * Append a caller-allocated command queue entry to the queue.
+ *
+ * The query itself must already have been put in the output buffer by the
+ * caller.
+ */
+static void
+pqAppendPipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ PGcommandQueueEntry **tail;
+
+ if (conn->cmd_queue_head == NULL)
+ tail = &conn->cmd_queue_head;
+ else
+ tail = &conn->cmd_queue_tail->next;
+
+ *tail = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * pqRecyclePipelineCmd
+ * Push a command queue entry onto the freelist. It must be an entry
+ * with null next pointer and not referenced by any other entry's next
+ * pointer.
+ */
+static void
+pqRecyclePipelineCmd(PGconn *conn, PGcommandQueueEntry *entry)
+{
+ if (entry == NULL)
+ return;
+
+ Assert(entry->next == NULL);
+
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
/* ====== accessor funcs for PGresult ======== */
@@ -3259,6 +3801,23 @@ PQflush(PGconn *conn)
return pqFlush(conn);
}
+/*
+ * pqPipelineFlush
+ *
+ * In pipeline mode, data will be flushed only when the out buffer reaches the
+ * threshold value. In non-pipeline mode, it behaves as stock pqFlush.
+ *
+ * Returns 0 on success.
+ */
+static int
+pqPipelineFlush(PGconn *conn)
+{
+ if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
+ (conn->outCount >= OUTBUFFER_THRESHOLD))
+ return (pqFlush(conn));
+ return 0;
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 6efa53d8b7..b46c6421f4 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -406,6 +406,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in pipeline mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index e4ee9d69d2..584922b340 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -177,14 +177,24 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
- else
- {
- pqInternalNotice(&conn->noticeHooks,
- "message type 0x%02x arrived from server while idle",
- id);
- /* Discard the unexpected message */
- conn->inCursor += msgLength;
- }
+
+ /*
+ * We're also IDLE when in pipeline mode we have completed
+ * processing the results of one query and are waiting for the
+ * next one in the pipeline. In this case, as above, just wait to
+ * see what's next.
+ */
+ if (conn->asyncStatus == PGASYNC_IDLE &&
+ conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->cmd_queue_head != NULL)
+ return;
+
+ /* Any other case is unexpected and we summarily skip it */
+ pqInternalNotice(&conn->noticeHooks,
+ "message type 0x%02x arrived from server while idle",
+ id);
+ /* Discard the unexpected message */
+ conn->inCursor += msgLength;
}
else
{
@@ -217,10 +227,28 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new
+ * query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF)
+ {
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_PIPELINE_SYNC);
+ if (!conn->result)
+ {
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ else
+ {
+ conn->pipelineStatus = PQ_PIPELINE_ON;
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -450,7 +478,7 @@ handleSyncLoss(PGconn *conn, char id, int msgLength)
id, msgLength);
/* build an error result holding the error message */
pqSaveErrorResult(conn);
- conn->asyncStatus = PGASYNC_READY; /* drop out of GetResult wait loop */
+ conn->asyncStatus = PGASYNC_READY; /* drop out of PQgetResult wait loop */
/* flush input data since we're giving up on processing it */
pqDropConnection(conn, true);
conn->status = CONNECTION_BAD; /* No more connection to backend */
@@ -875,6 +903,10 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ /* If in pipeline mode, set error indicator for it */
+ if (isError && conn->pipelineStatus != PQ_PIPELINE_OFF)
+ conn->pipelineStatus = PQ_PIPELINE_ABORTED;
+
/*
* If this is an error message, pre-emptively clear any incomplete query
* result we may have. We'd just throw it away below anyway, and
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 47a098b4b9..fa40556f60 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -96,7 +96,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_PIPELINE_SYNC, /* pipeline synchronization point */
+ PGRES_PIPELINE_ABORTED, /* Command didn't run because of an abort
+ * earlier in a pipeline */
} ExecStatusType;
typedef enum
@@ -136,6 +139,16 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PGpipelineStatus - Current status of pipeline mode
+ */
+typedef enum
+{
+ PQ_PIPELINE_OFF,
+ PQ_PIPELINE_ON,
+ PQ_PIPELINE_ABORTED
+} PGpipelineStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -327,6 +340,7 @@ extern int PQserverVersion(const PGconn *conn);
extern char *PQerrorMessage(const PGconn *conn);
extern int PQsocket(const PGconn *conn);
extern int PQbackendPID(const PGconn *conn);
+extern PGpipelineStatus PQpipelineStatus(const PGconn *conn);
extern int PQconnectionNeedsPassword(const PGconn *conn);
extern int PQconnectionUsedPassword(const PGconn *conn);
extern int PQclientEncoding(const PGconn *conn);
@@ -434,6 +448,11 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for pipeline mode management */
+extern int PQenterPipelineMode(PGconn *conn);
+extern int PQexitPipelineMode(PGconn *conn);
+extern int PQsendPipeline(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 0c9e95f1a7..f0ccd8c0f6 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,7 +217,11 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, more results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
@@ -229,7 +233,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* Sync at end of a pipeline */
} PGQueryClass;
/* Target server type (decoded value of target_session_attrs) */
@@ -321,6 +326,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by pipeline mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct PGcommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct PGcommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about each of possibly several hosts
* mentioned in the connection string. Most fields are derived by splitting
@@ -413,6 +434,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PGpipelineStatus pipelineStatus; /* status of pipeline mode */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY OUT */
@@ -425,6 +447,16 @@ struct pg_conn
pg_conn_host *connhost; /* details about each named host */
char *connip; /* IP address for current network connection */
+ /*
+ * The command queue, for pipeline mode.
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -824,6 +856,11 @@ extern ssize_t pg_GSS_read(PGconn *conn, void *ptr, size_t len);
*/
#define pqIsnonblocking(conn) ((conn)->nonblocking)
+/*
+ * Connection's outbuffer threshold.
+ */
+#define OUTBUFFER_THRESHOLD 65536
+
#ifdef ENABLE_NLS
extern char *libpq_gettext(const char *msgid) pg_attribute_format_arg(1);
extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigned long n) pg_attribute_format_arg(1) pg_attribute_format_arg(2);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 5391f461a2..6e31458be2 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
test_extensions \
test_ginpostinglist \
test_integerset \
+ test_libpq \
test_misc \
test_parser \
test_pg_dump \
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000000..4fbf97a5b0
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/pipeline
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000000..64a3a4af5e
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/test_libpq/Makefile
+
+PROGRAM = pipeline
+OBJS = pipeline.o
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000000..d8174dd579
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/pipeline.c b/src/test/modules/test_libpq/pipeline.c
new file mode 100644
index 0000000000..f4a2bdec57
--- /dev/null
+++ b/src/test/modules/test_libpq/pipeline.c
@@ -0,0 +1,1077 @@
+/*
+ * src/test/modules/test_libpq/pipeline.c
+ * Verify libpq pipeline execution functionality
+ */
+#include "postgres_fe.h"
+
+#include <sys/time.h>
+
+#include "catalog/pg_type_d.h"
+#include "common/fe_memutils.h"
+#include "libpq-fe.h"
+#include "portability/instr_time.h"
+
+
+static void exit_nicely(PGconn *conn);
+
+const char *const progname = "pipeline";
+
+
+#define DEBUG
+#ifdef DEBUG
+#define pg_debug(...) do { fprintf(stderr, __VA_ARGS__); } while (0)
+#else
+#define pg_debug(...)
+#endif
+
+static const char *const drop_table_sql =
+"DROP TABLE IF EXISTS pq_pipeline_demo";
+static const char *const create_table_sql =
+"CREATE UNLOGGED TABLE pq_pipeline_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql =
+"INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+/*
+ * Print an error to stderr and terminate the program.
+ */
+#define pg_fatal(...) pg_fatal_impl(__LINE__, __VA_ARGS__)
+static void
+pg_fatal_impl(int line, const char *fmt,...)
+{
+ va_list args;
+
+ fprintf(stderr, "\n"); /* XXX hack */
+ fprintf(stderr, "%s:%d: ", progname, line);
+
+ va_start(args, fmt);
+ vfprintf(stderr, fmt, args);
+ va_end(args);
+ printf("Failure, exiting\n");
+ exit(1);
+}
+
+static void
+test_disallowed(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("Unable to enter pipeline mode\n");
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not activated properly\n");
+
+ /* PQexec should fail in pipeline mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("PQexec should fail in pipeline mode but succeeded\n");
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ pg_fatal("PQsendQuery should fail in pipeline mode but succeeded\n");
+
+ /* Entering pipeline mode when already in pipeline mode is OK */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("re-entering pipeline mode should be a no-op but failed\n");
+
+ if (PQisBusy(conn) != 0)
+ pg_fatal("PQisBusy should return 0 when idle in pipeline mode, returned 1\n");
+
+ /* ok, back to normal command mode */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("couldn't exit idle empty pipeline mode\n");
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Pipeline mode not terminated properly\n");
+
+ /* exiting pipeline mode when not in pipeline mode should be a no-op */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("pipeline mode exit when not in pipeline mode should succeed but failed\n");
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("PQexec should succeed after exiting pipeline mode but failed with: %s\n",
+ PQerrorMessage(conn));
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_simple_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple pipeline... ");
+
+ /*
+ * Enter pipeline mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ pg_fatal("Expected blocking connection mode\n");
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode with work in progress should fail, but succeeded\n");
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending pipeline failed: %s\n", PQerrorMessage(conn));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first query result.\n");
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit pipeline mode yet
+ */
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result PGRES_PIPELINE_SYNC expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of PGRES_PIPELINE_SYNC, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after pipeline end: %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* ... until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("Exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+static void
+test_multi_pipelines(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi pipeline... ");
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /* OK, start processing the results */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from first pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ pg_fatal("PQgetResult returned something extra after first result\n");
+
+ if (PQexitPipelineMode(conn) != 0)
+ pg_fatal("exiting pipeline mode after query but before sync succeeded incorrectly\n");
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when sync result expected: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+
+ PQclear(res);
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* second pipeline */
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Unexpected result code %s from second pipeline item\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("PQgetResult returned null when there's a pipeline item: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* We're still in pipeline mode ... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+}
+
+/*
+ * When an operation in a pipeline fails the rest of the pipeline is flushed. We
+ * still have to get results for each pipeline item, but the item will just be
+ * a PGRES_PIPELINE_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the pipeline. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_aborted_pipeline(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted pipeline... ");
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * Queue up a couple of small pipelines and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * pipeline ERRORs.
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "1";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching first insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn, "SELECT no_such_function($1)",
+ 1, dummy_param_oids, dummy_params,
+ NULL, NULL, 0) != 1)
+ pg_fatal("dispatching error select failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "2";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second insert failed: %s\n", PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Sending first pipeline failed: %s\n", PQerrorMessage(conn));
+
+ dummy_params[0] = "3";
+ if (PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching second-pipeline insert failed: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendPipeline(conn) != 1)
+ pg_fatal("Ending second pipeline failed: %s\n", PQerrorMessage(conn));
+
+ /*
+ * OK, start processing the pipeline results.
+ *
+ * We should get a command-ok for the first query, then a fatal error and
+ * a pipeline aborted message for the second insert, a pipeline-end, then
+ * a command-ok and a pipeline-ok for the second pipeline operation.
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result status %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Second query caused error, so we expect an error next */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ pg_fatal("Unexpected result code -- expected PGRES_FATAL_ERROR, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+
+ /*
+ * pipeline should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third pipeline since we've already sent a
+ * second. The aborted flag relates only to the pipeline being received.
+ */
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* third query in pipeline, the second insert */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_ABORTED)
+ pg_fatal("Unexpected result code -- expected PGRES_PIPELINE_ABORTED, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* NULL result to signal end-of-results for this command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_ABORTED)
+ pg_fatal("pipeline should be flagged as aborted but isn't\n");
+
+ /* Ensure we're still in pipeline */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /*
+ * The end of a failed pipeline is a PGRES_PIPELINE_SYNC.
+ *
+ * (This is so clients know to start processing results normally again and
+ * can tell the difference between skipped commands and the sync.)
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code from first pipeline sync\n"
+ "Expected PGRES_PIPELINE_SYNC, got %s\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* after the synchronization point we get a NULL result */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_ABORTED)
+ pg_fatal("sync should've cleared the aborted flag but didn't\n");
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* the insert from the second pipeline */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("Unexpected result code %s from first item in second pipeline\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ /* Read the NULL result at the end of the command */
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s\n", PQresStatus(PQresultStatus(res)));
+
+ /* the second pipeline sync */
+ if ((res = PQgetResult(conn)) == NULL)
+ pg_fatal("Unexpected NULL result: %s\n", PQerrorMessage(conn));
+ if (PQresultStatus(res) != PGRES_PIPELINE_SYNC)
+ pg_fatal("Unexpected result code %s from second pipeline sync\n",
+ PQresStatus(PQresultStatus(res)));
+ PQclear(res);
+
+ if ((res = PQgetResult(conn)) != NULL)
+ pg_fatal("Expected null result, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQerrorMessage(conn));
+
+ /* We're still in pipeline mode... */
+ if (PQpipelineStatus(conn) == PQ_PIPELINE_OFF)
+ pg_fatal("Fell out of pipeline mode somehow\n");
+
+ /* until we end it, which we can safely do now */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQpipelineStatus(conn) != PQ_PIPELINE_OFF)
+ pg_fatal("exiting pipeline mode didn't seem to work\n");
+
+ fprintf(stderr, "ok\n");
+
+ /*-
+ * Since we fired the pipelines off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st pipeline
+ * - First insert applied
+ * - Second statement aborted xact
+ * - Third insert skipped
+ * - Sync rolled back first implicit xact
+ * - Implicit xact created by server around 2nd pipeline
+ * - insert applied from 2nd pipeline
+ * - Sync commits 2nd xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM pq_pipeline_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ pg_fatal("Expected tuples, got %s: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ if (PQntuples(res) != 1)
+ pg_fatal("expected 1 result, got %d\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ pg_fatal("expected only insert with value 3, got %s", val);
+ }
+
+ PQclear(res);
+}
+
+/* State machine enum for test_pipelined_insert */
+typedef enum PipelineInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} PipelineInsertStep;
+
+static void
+test_pipelined_insert(PGconn *conn, int n_rows)
+{
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ PipelineInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a pipelined insert into a table created at the start of the pipeline
+ */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n", PQerrorMessage(conn));
+
+ while (send_step != BI_PREPARE)
+ {
+ const char *sql;
+
+ switch (send_step)
+ {
+ case BI_BEGIN_TX:
+ sql = "BEGIN TRANSACTION";
+ send_step = BI_DROP_TABLE;
+ break;
+
+ case BI_DROP_TABLE:
+ sql = drop_table_sql;
+ send_step = BI_CREATE_TABLE;
+ break;
+
+ case BI_CREATE_TABLE:
+ sql = create_table_sql;
+ send_step = BI_PREPARE;
+ break;
+
+ default:
+ pg_fatal("invalid state");
+ }
+
+ pg_debug("sending: %s\n", sql);
+ if (PQsendQueryParams(conn, sql,
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("dispatching %s failed: %s\n", sql, PQerrorMessage(conn));
+ }
+
+ Assert(send_step == BI_PREPARE);
+ pg_debug("sending: %s\n", insert_sql);
+ if (PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids) != 1)
+ pg_fatal("dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the pipeline before processing
+ * results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ pg_fatal("failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ PGresult *res;
+ const char *cmdtag;
+ const char *description = "";
+ int status;
+
+ /*
+ * Read next result. If no more results from this query,
+ * advance to the next query
+ */
+ res = PQgetResult(conn);
+ if (res == NULL)
+ continue;
+
+ status = PGRES_COMMAND_OK;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ recv_step++;
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ recv_step++;
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ recv_step++;
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ recv_step++;
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive == 0)
+ recv_step++;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ recv_step++;
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_PIPELINE_SYNC;
+ recv_step++;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ description = "";
+ abort();
+ }
+
+ if (PQresultStatus(res) != status)
+ pg_fatal("%s reported status %s, expected %s\n"
+ "Error message: \"%s\"\n",
+ description, PQresStatus(PQresultStatus(res)),
+ PQresStatus(status), PQerrorMessage(conn));
+
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ pg_fatal("%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+
+ pg_debug("Got %s OK\n", cmdtag[0] != '\0' ? cmdtag : description);
+
+ PQclear(res);
+ }
+ }
+
+ /* Write more rows and/or the end pipeline message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0) == 1)
+ {
+ pg_debug("sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQsendPipeline(conn) == 1)
+ {
+ fprintf(stdout, "Sent pipeline\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending pipeline failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+ }
+
+ /* We've got the sync message and the pipeline should be done */
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("attempt to exit pipeline mode failed when it should've succeeded: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ pg_fatal("failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+}
+
+static void
+test_singlerowmode(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ bool pipeline_ended = false;
+
+ /* 1 pipeline, 3 queries in it */
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ for (i = 0; i < 3; i++)
+ {
+ char *param[1];
+
+ param[0] = psprintf("%d", 44 + i);
+
+ if (PQsendQueryParams(conn,
+ "SELECT generate_series(42, $1)",
+ 1,
+ NULL,
+ (const char **) param,
+ NULL,
+ NULL,
+ 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ pfree(param[0]);
+ }
+ PQsendPipeline(conn);
+
+ for (i = 0; !pipeline_ended; i++)
+ {
+ bool first = true;
+ bool saw_ending_tuplesok;
+ bool isSingleTuple = false;
+
+ /* Set single row mode for only first 2 SELECT queries */
+ if (i < 2)
+ {
+ if (PQsetSingleRowMode(conn) != 1)
+ pg_fatal("PQsetSingleRowMode() failed for i=%d\n", i);
+ }
+
+ /* Consume rows for this query */
+ saw_ending_tuplesok = false;
+ while ((res = PQgetResult(conn)) != NULL)
+ {
+ ExecStatusType est = PQresultStatus(res);
+
+ if (est == PGRES_PIPELINE_SYNC)
+ {
+ fprintf(stderr, "end of pipeline reached\n");
+ pipeline_ended = true;
+ PQclear(res);
+ if (i != 3)
+ pg_fatal("Expected three results, got %d\n", i);
+ break;
+ }
+
+ /* Expect SINGLE_TUPLE for queries 0 and 1, TUPLES_OK for 2 */
+ if (first)
+ {
+ if (i <= 1 && est != PGRES_SINGLE_TUPLE)
+ pg_fatal("Expected PGRES_SINGLE_TUPLE for query %d, got %s\n",
+ i, PQresStatus(est));
+ if (i >= 2 && est != PGRES_TUPLES_OK)
+ pg_fatal("Expected PGRES_TUPLES_OK for query %d, got %s\n",
+ i, PQresStatus(est));
+ first = false;
+ }
+
+ fprintf(stderr, "Result status %s for query %d", PQresStatus(est), i);
+ switch (est)
+ {
+ case PGRES_TUPLES_OK:
+ fprintf(stderr, ", tuples: %d\n", PQntuples(res));
+ saw_ending_tuplesok = true;
+ if (isSingleTuple)
+ {
+ if (PQntuples(res) == 0)
+ fprintf(stderr, "all tuples received in query %d\n", i);
+ else
+ pg_fatal("Expected to follow PGRES_SINGLE_TUPLE, "
+ "but received PGRES_TUPLES_OK directly instead\n");
+ }
+ break;
+
+ case PGRES_SINGLE_TUPLE:
+ fprintf(stderr, ", %d tuple: %s\n", PQntuples(res), PQgetvalue(res, 0, 0));
+ break;
+
+ default:
+ pg_fatal("unexpected\n");
+ }
+ PQclear(res);
+ }
+ if (!pipeline_ended && !saw_ending_tuplesok)
+ pg_fatal("didn't get expected terminating TUPLES_OK\n");
+ }
+
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+}
+
+/*
+ * Simple test to verify that a pipeline is discarded as a whole when there's
+ * an error, ignoring transaction commands.
+ */
+static void
+test_transaction(PGconn *conn)
+{
+ int i;
+ PGresult *res;
+ int num_sends = 0;
+
+ res = PQexec(conn, "DROP TABLE IF EXISTS pq_pipeline_tst;"
+ "CREATE TABLE pq_pipeline_tst (id int)");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("failed to create test table: %s",
+ PQerrorMessage(conn));
+ PQclear(res);
+
+ if (PQenterPipelineMode(conn) != 1)
+ pg_fatal("failed to enter pipeline mode: %s\n",
+ PQerrorMessage(conn));
+
+ if (PQsendQueryParams(conn,
+ "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "SELECT 0/0",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "/* first */ ROLLBACK",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (1)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (2)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ if (PQsendQueryParams(conn,
+ "/* second */ ROLLBACK",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ if (PQsendQueryParams(conn,
+ "INSERT INTO pq_pipeline_tst VALUES (3)",
+ 0, NULL, NULL, NULL, NULL, 0) != 1)
+ pg_fatal("failed to send query: %s\n",
+ PQerrorMessage(conn));
+ PQsendPipeline(conn);
+ num_sends++;
+
+ for (i = 0; i < 20; i++)
+ {
+ ExecStatusType restype;
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "got NULL result\n");
+ continue;
+ }
+ restype = PQresultStatus(res);
+ fprintf(stderr, "%d: got status %s", i, PQresStatus(restype));
+ if (restype == PGRES_FATAL_ERROR)
+ fprintf(stderr, "; error: %s", PQerrorMessage(conn));
+ else if (restype == PGRES_PIPELINE_ABORTED)
+ {
+ fprintf(stderr, ": command didn't run because pipeline aborted\n");
+ }
+ else
+ fprintf(stderr, "\n");
+ PQclear(res);
+
+ if (restype == PGRES_PIPELINE_SYNC)
+ num_sends--;
+ if (num_sends <= 0)
+ break;
+ }
+ if (PQexitPipelineMode(conn) != 1)
+ pg_fatal("failed to end pipeline mode: %s\n", PQerrorMessage(conn));
+
+ res = PQexec(conn, "SELECT * FROM pq_pipeline_tst");
+ printf("got %d tuples:\n", PQntuples(res));
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ printf("tuple: %s\n", PQgetvalue(res, i, 0));
+ }
+ PQclear(res);
+}
+
+static void
+usage(const char *progname)
+{
+ fprintf(stderr, "%s tests libpq's pipeline mode.\n\n", progname);
+ fprintf(stderr, "Usage:\n");
+ fprintf(stderr, " %s testname [conninfo [number_of_rows]]\n", progname);
+ fprintf(stderr, "Tests:\n");
+ fprintf(stderr, " disallowed_in_pipeline\n");
+ fprintf(stderr, " simple_pipeline\n");
+ fprintf(stderr, " multi_pipeline\n");
+ fprintf(stderr, " pipeline_abort\n");
+ fprintf(stderr, " singlerow\n");
+ fprintf(stderr, " pipeline_insert\n");
+ fprintf(stderr, " transaction\n");
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo = "";
+ PGconn *conn;
+ int numrows = 10000;
+
+ /*
+ * The testname parameter is mandatory; it can be followed by a conninfo
+ * string and number of rows.
+ */
+ if (argc < 2 || argc > 4)
+ {
+ usage(argv[0]);
+ exit(1);
+ }
+
+ if (argc >= 3)
+ conninfo = pg_strdup(argv[2]);
+
+ if (argc >= 4)
+ {
+ errno = 0;
+ numrows = strtol(argv[3], NULL, 10);
+ if (errno != 0 || numrows <= 0)
+ {
+ fprintf(stderr, "couldn't parse \"%s\" as a positive integer\n", argv[3]);
+ exit(1);
+ }
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (strcmp(argv[1], "disallowed_in_pipeline") == 0)
+ test_disallowed(conn);
+ else if (strcmp(argv[1], "simple_pipeline") == 0)
+ test_simple_pipeline(conn);
+ else if (strcmp(argv[1], "multi_pipeline") == 0)
+ test_multi_pipelines(conn);
+ else if (strcmp(argv[1], "pipeline_abort") == 0)
+ test_aborted_pipeline(conn);
+ else if (strcmp(argv[1], "pipeline_insert") == 0)
+ test_pipelined_insert(conn, numrows);
+ else if (strcmp(argv[1], "singlerow") == 0)
+ test_singlerowmode(conn);
+ else if (strcmp(argv[1], "transaction") == 0)
+ test_transaction(conn);
+ else
+ {
+ fprintf(stderr, "\"%s\" is not a recognized test name\n", argv[1]);
+ usage(argv[0]);
+ exit(1);
+ }
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+ return 0;
+}
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000000..211629db2d
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,30 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $numrows = 10000;
+my @tests =
+ qw(disallowed_in_pipeline
+ simple_pipeline
+ multi_pipeline
+ pipeline_abort
+ pipeline_insert
+ singlerow);
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(
+ [ 'pipeline', $testname, $node->connstr('postgres'), $numrows ],
+ "pipeline $testname");
+}
+
+$node->stop('fast');
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8bd95aefa1..1bd23eec6d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1561,10 +1561,12 @@ PG_Locale_Strategy
PG_Lock_Status
PG_init_t
PGcancel
+PGcommandQueueEntry
PGconn
PGdataValue
PGlobjfuncs
PGnotify
+PGpipelineStatus
PGresAttDesc
PGresAttValue
PGresParamDesc
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-03-03 07:07 ` RE: PATCH: Batch/pipelining support for libpq k.jamison@fujitsu.com <k.jamison@fujitsu.com>
2021-03-03 16:34 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
@ 2021-03-03 23:42 ` 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-04 00:24 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
1 sibling, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-03 23:42 UTC (permalink / raw)
To: k.jamison@fujitsu.com <k.jamison@fujitsu.com>; +Cc: Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
On 2021-Mar-03, 'Alvaro Herrera' wrote:
> I'm much more comfortable with this version, so I'm marking the patch as
> Ready for Committer in case anybody else wants to look at this before I
> push it.
Actually, I just noticed a pretty serious problem, which is that when we
report an error, we don't set "conn->errQuery" to the current query but
to the *previous* query, leading to non-sensical error reports like
$ ./pipeline pipeline_abort
aborted pipeline... ERROR: no existe la función no_such_function(integer)
LINE 1: INSERT INTO pq_pipeline_demo(itemno) VALUES ($1);
^
HINT: Ninguna función coincide en el nombre y tipos de argumentos. Puede ser necesario agregar conversión explícita de tipos.
ok
(Sorry about the Spanish there, but the gist of the problem is that
we're positioning the error cursor on a query that succeeded prior to
the query that raised the error.)
This should obviously not occur. I'm trying to figure out how to repair
it and not break everything again ...
--
Álvaro Herrera Valdivia, Chile
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-03-03 07:07 ` RE: PATCH: Batch/pipelining support for libpq k.jamison@fujitsu.com <k.jamison@fujitsu.com>
2021-03-03 16:34 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-03 23:42 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
@ 2021-03-04 00:24 ` 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-04 01:51 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-04 00:24 UTC (permalink / raw)
To: k.jamison@fujitsu.com <k.jamison@fujitsu.com>; +Cc: Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
On 2021-Mar-03, 'Alvaro Herrera' wrote:
> This should obviously not occur. I'm trying to figure out how to repair
> it and not break everything again ...
I think trying to set up the connection state so that the next query
appears in conn->last_query prior to PQgetResult being called again
leads to worse breakage. The simplest fix seems to make fe-protocol3.c
aware that in this case, the query is in conn->cmd_queue_head instead,
as in the attached patch.
--
Álvaro Herrera 39°49'30"S 73°17'W
Attachments:
error-fix.patch (text/x-diff, 1.1K) download
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 584922b340..b89f500902 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -962,9 +962,21 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
* Save the active query text, if any, into res as well; but only if we
* might need it for an error cursor display, which is only true if there
* is a PG_DIAG_STATEMENT_POSITION field.
+ *
+ * Note that in pipeline mode, we have not yet advanced the query pointer
+ * to the next query, so we have to look at that.
*/
- if (have_position && conn->last_query && res)
- res->errQuery = pqResultStrdup(res, conn->last_query);
+ if (have_position && res)
+ {
+ if (conn->pipelineStatus != PQ_PIPELINE_OFF &&
+ conn->asyncStatus == PGASYNC_IDLE)
+ {
+ if (conn->cmd_queue_head)
+ res->errQuery = pqResultStrdup(res, conn->cmd_queue_head->query);
+ }
+ else if (conn->last_query)
+ res->errQuery = pqResultStrdup(res, conn->last_query);
+ }
/*
* Now build the "overall" error message for PQresultErrorMessage.
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-03-03 07:07 ` RE: PATCH: Batch/pipelining support for libpq k.jamison@fujitsu.com <k.jamison@fujitsu.com>
2021-03-03 16:34 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-03 23:42 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-04 00:24 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
@ 2021-03-04 01:51 ` 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-08 23:02 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-04 01:51 UTC (permalink / raw)
To: k.jamison@fujitsu.com <k.jamison@fujitsu.com>; +Cc: Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
On 2021-Mar-03, 'Alvaro Herrera' wrote:
> On 2021-Mar-03, 'Alvaro Herrera' wrote:
>
> > This should obviously not occur. I'm trying to figure out how to repair
> > it and not break everything again ...
>
> I think trying to set up the connection state so that the next query
> appears in conn->last_query prior to PQgetResult being called again
> leads to worse breakage. The simplest fix seems to make fe-protocol3.c
> aware that in this case, the query is in conn->cmd_queue_head instead,
> as in the attached patch.
I wonder if it would make sense to get rid of conn->last_query
completely and just rely always on conn->cmd_queue_head, where the
normal (non-pipeline) would use the first entry of the command queue.
That might end up being simpler than pipeline mode "pretending" to take
over ->last_query.
--
Álvaro Herrera 39°49'30"S 73°17'W
"La verdad no siempre es bonita, pero el hambre de ella sí"
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-03-03 07:07 ` RE: PATCH: Batch/pipelining support for libpq k.jamison@fujitsu.com <k.jamison@fujitsu.com>
2021-03-03 16:34 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-03 23:42 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-04 00:24 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-04 01:51 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
@ 2021-03-08 23:02 ` 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
0 siblings, 0 replies; 182+ messages in thread
From: Alvaro Herrera @ 2021-03-08 23:02 UTC (permalink / raw)
To: k.jamison@fujitsu.com <k.jamison@fujitsu.com>; +Cc: Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL-development <pgsql-hackers@lists.postgresql.org>
On 2021-Mar-03, 'Alvaro Herrera' wrote:
> I wonder if it would make sense to get rid of conn->last_query
> completely and just rely always on conn->cmd_queue_head, where the
> normal (non-pipeline) would use the first entry of the command queue.
> That might end up being simpler than pipeline mode "pretending" to take
> over ->last_query.
I experimented with this today and it appears to be a good change,
though I haven't been able to make everything work correctly yet.
--
Álvaro Herrera Valdivia, Chile
"Entristecido, Wutra (canción de Las Barreras)
echa a Freyr a rodar
y a nosotros al mar"
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-03-03 07:07 ` RE: PATCH: Batch/pipelining support for libpq k.jamison@fujitsu.com <k.jamison@fujitsu.com>
2021-03-03 16:34 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
@ 2021-03-04 00:45 ` Justin Pryzby <pryzby@telsasoft.com>
2021-03-08 22:37 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
1 sibling, 1 reply; 182+ messages in thread
From: Justin Pryzby @ 2021-03-04 00:45 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: k.jamison@fujitsu.com, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; pgsql-hackers@lists.postgresql.org
I'm proposing some minor changes.
--
Justin
Attachments:
0002-doc-review.notapxtch (text/x-diff, 12.2K) download
From 35ed4dc1fc770834972396f7eeed142f6dabee88 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Wed, 3 Mar 2021 16:36:40 -0600
Subject: [PATCH 2/2] doc review
---
doc/src/sgml/libpq.sgml | 67 +++++++++++++-------------
src/interfaces/libpq/fe-exec.c | 2 +-
src/test/modules/test_libpq/pipeline.c | 6 +--
3 files changed, 38 insertions(+), 37 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index a34ecbb144..e2865a218b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -3130,8 +3130,9 @@ ExecStatusType PQresultStatus(const PGresult *res);
<term><literal>PGRES_PIPELINE_SYNC</literal></term>
<listitem>
<para>
- The <structname>PGresult</structname> represents a point in a
- pipeline where a synchronization point has been established.
+ The <structname>PGresult</structname> represents a
+ synchronization point in pipeline mode, requested by
+ <xref linkend="libpq-PQsendPipeline"/>.
This status occurs only when pipeline mode has been selected.
</para>
</listitem>
@@ -3141,9 +3142,9 @@ ExecStatusType PQresultStatus(const PGresult *res);
<term><literal>PGRES_PIPELINE_ABORTED</literal></term>
<listitem>
<para>
- The <structname>PGresult</structname> represents a pipeline that's
+ The <structname>PGresult</structname> represents a pipeline that has
received an error from the server. <function>PQgetResult</function>
- must be called repeatedly, and it will return this status code,
+ must be called repeatedly, and each time it will return this status code
until the end of the current pipeline, at which point it will return
<literal>PGRES_PIPELINE_SYNC</literal> and normal processing can
resume.
@@ -4953,7 +4954,7 @@ int PQflush(PGconn *conn);
<title>Using Pipeline Mode</title>
<para>
- To issue pipelines the application must switch a connection into pipeline mode.
+ To issue pipelines, the application must switch a connection into pipeline mode.
Enter pipeline mode with <xref linkend="libpq-PQenterPipelineMode"/>
or test whether pipeline mode is active with
<xref linkend="libpq-PQpipelineStatus"/>.
@@ -4975,7 +4976,7 @@ int PQflush(PGconn *conn);
server will block trying to send results to the client from queries
it has already processed. This only occurs when the client sends
enough queries to fill both its output buffer and the server's receive
- buffer before switching to processing input from the server,
+ buffer before it switches to processing input from the server,
but it's hard to predict exactly when that will happen.
</para>
</footnote>
@@ -5025,7 +5026,7 @@ int PQflush(PGconn *conn);
<title>Processing Results</title>
<para>
- To process the result of one pipeline query, the application calls
+ To process the result of one query in a pipeline, the application calls
<function>PQgetResult</function> repeatedly and handles each result
until <function>PQgetResult</function> returns null.
The result from the next query in the pipeline may then be retrieved using
@@ -5057,10 +5058,10 @@ int PQflush(PGconn *conn);
<type>PGresult</type> types <literal>PGRES_PIPELINE_SYNC</literal>
and <literal>PGRES_PIPELINE_ABORTED</literal>.
<literal>PGRES_PIPELINE_SYNC</literal> is reported exactly once for each
- <function>PQsendPipeline</function> call at the corresponding point in
- the result stream.
+ <function>PQsendPipeline</function> after retrieving results for all
+ queries in the pipeline.
<literal>PGRES_PIPELINE_ABORTED</literal> is emitted in place of a normal
- result stream result for the first error and all subsequent results
+ stream result for the first error and all subsequent results
except <literal>PGRES_PIPELINE_SYNC</literal> and null;
see <xref linkend="libpq-pipeline-errors"/>.
</para>
@@ -5075,7 +5076,8 @@ int PQflush(PGconn *conn);
application about the query currently being processed (except that
<function>PQgetResult</function> returns null to indicate that we start
returning the results of next query). The application must keep track
- of the order in which it sent queries and the expected results.
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
Applications will typically use a state machine or a FIFO queue for this.
</para>
@@ -5091,10 +5093,10 @@ int PQflush(PGconn *conn);
</para>
<para>
- From the client perspective, after the client gets a
- <literal>PGRES_FATAL_ERROR</literal> return from
- <function>PQresultStatus</function> the pipeline is flagged as aborted.
- <application>libpq</application> will report
+ From the client perspective, after <function>PQresultStatus</function>
+ returns <literal>PGRES_FATAL_ERROR</literal>,
+ the pipeline is flagged as aborted.
+ <function>PQresultStatus</function>, will report a
<literal>PGRES_PIPELINE_ABORTED</literal> result for each remaining queued
operation in an aborted pipeline. The result for
<function>PQsendPipeline</function> is reported as
@@ -5108,8 +5110,8 @@ int PQflush(PGconn *conn);
</para>
<para>
- If the pipeline used an implicit transaction then operations that have
- already executed are rolled back and operations that were queued for after
+ If the pipeline used an implicit transaction, then operations that have
+ already executed are rolled back and operations that were queued to follow
the failed operation are skipped entirely. The same behaviour holds if the
pipeline starts and commits a single explicit transaction (i.e. the first
statement is <literal>BEGIN</literal> and the last is
@@ -5145,11 +5147,11 @@ int PQflush(PGconn *conn);
<para>
The client application should generally maintain a queue of work
- still to be dispatched and a queue of work that has been dispatched
+ remaining to be dispatched and a queue of work that has been dispatched
but not yet had its results processed. When the socket is writable
it should dispatch more work. When the socket is readable it should
read results and process them, matching them up to the next entry in
- its expected results queue. Based on available memory, results from
+ its expected results queue. Based on available memory, results from the
socket should be read frequently: there's no need to wait until the
pipeline end to read the results. Pipelines should be scoped to logical
units of work, usually (but not necessarily) one transaction per pipeline.
@@ -5191,8 +5193,8 @@ int PQflush(PGconn *conn);
<listitem>
<para>
- Returns current pipeline mode status of the <application>libpq</application>
- connection.
+ Returns the current pipeline mode status of the
+ <application>libpq</application> connection.
<synopsis>
PGpipelineStatus PQpipelineStatus(const PGconn *conn);
</synopsis>
@@ -5233,11 +5235,10 @@ PGpipelineStatus PQpipelineStatus(const PGconn *conn);
<listitem>
<para>
The <application>libpq</application> connection is in pipeline
- mode and an error has occurred while processing the current
- pipeline.
- The aborted flag is cleared as soon as the result
- of the <function>PQsendPipeline</function> at the end of the aborted
- pipeline is processed. Clients don't usually need this function to
+ mode and an error occurred while processing the current pipeline.
+ The aborted flag is cleared when <function>PQresultStatus</function>
+ returns PGRES_PIPELINE_SYNC at the end of the pipeline.
+ Clients don't usually need this function to
verify aborted status, as they can tell that the pipeline is aborted
from the <literal>PGRES_PIPELINE_ABORTED</literal> result code.
</para>
@@ -5328,7 +5329,7 @@ int PQsendPipeline(PGconn *conn);
Returns 1 for success. Returns 0 if the connection is not in
pipeline mode or sending a
<link linkend="protocol-flow-ext-query">sync message</link>
- is failed.
+ failed.
</para>
</listitem>
</varlistentry>
@@ -5349,7 +5350,7 @@ int PQsendPipeline(PGconn *conn);
<para>
Pipeline mode is most useful when the server is distant, i.e., network latency
(<quote>ping time</quote>) is high, and also when many small operations
- are being performed in rapid sequence. There is usually less benefit
+ are being performed in rapid succession. There is usually less benefit
in using pipelined commands when each query takes many multiples of the client/server
round-trip time to execute. A 100-statement operation run on a server
300ms round-trip-time away would take 30 seconds in network latency alone
@@ -5367,7 +5368,7 @@ int PQsendPipeline(PGconn *conn);
<para>
Pipeline mode is not useful when information from one operation is required by
the client to produce the next operation. In such cases, the client
- must introduce a synchronization point and wait for a full client/server
+ would have to introduce a synchronization point and wait for a full client/server
round-trip to get the results it needs. However, it's often possible to
adjust the client design to exchange the required information server-side.
Read-modify-write cycles are especially good candidates; for example:
@@ -5392,10 +5393,10 @@ UPDATE mytable SET x = x + 1 WHERE id = 42;
<note>
<para>
- The pipeline API was introduced in PostgreSQL 14, but clients using
- the PostgreSQL 14 version of <application>libpq</application> can use
- pipelines on server versions 7.4 and newer. Pipeline mode works on any server
- that supports the v3 extended query protocol.
+ The pipeline API was introduced in <productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
</para>
</note>
</sect2>
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 7ae7c14948..d7b036a35c 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -3803,7 +3803,7 @@ pqPipelineFlush(PGconn *conn)
{
if ((conn->pipelineStatus == PQ_PIPELINE_OFF) ||
(conn->outCount >= OUTBUFFER_THRESHOLD))
- return (pqFlush(conn));
+ return pqFlush(conn);
return 0;
}
diff --git a/src/test/modules/test_libpq/pipeline.c b/src/test/modules/test_libpq/pipeline.c
index f4a2bdec57..01d5a9a8ff 100644
--- a/src/test/modules/test_libpq/pipeline.c
+++ b/src/test/modules/test_libpq/pipeline.c
@@ -126,7 +126,7 @@ test_simple_pipeline(PGconn *conn)
* process the results of as they come in.
*
* For a simple case we should be able to do this without interim
- * processing of results since our out buffer will give us enough slush to
+ * processing of results since our output buffer will give us enough slush to
* work with and we won't block on sending. So blocking mode is fine.
*/
if (PQisnonblocking(conn))
@@ -603,7 +603,7 @@ test_pipelined_insert(PGconn *conn, int n_rows)
/*
* Now we start inserting. We'll be sending enough data that we could fill
- * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * our output buffer, so to avoid deadlocking we need to enter nonblocking
* mode and consume input while we send more output. As results of each
* query are processed we should pop them to allow processing of the next
* query. There's no need to finish the pipeline before processing
@@ -635,7 +635,7 @@ test_pipelined_insert(PGconn *conn, int n_rows)
}
/*
- * Process any results, so we keep the server's out buffer free
+ * Process any results, so we keep the server's output buffer free
* flowing and it can continue to process input
*/
if (FD_ISSET(sock, &input_mask))
--
2.17.0
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Re: PATCH: Batch/pipelining support for libpq Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-03-03 07:07 ` RE: PATCH: Batch/pipelining support for libpq k.jamison@fujitsu.com <k.jamison@fujitsu.com>
2021-03-03 16:34 ` Re: PATCH: Batch/pipelining support for libpq 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-04 00:45 ` Re: PATCH: Batch/pipelining support for libpq Justin Pryzby <pryzby@telsasoft.com>
@ 2021-03-08 22:37 ` David G. Johnston <david.g.johnston@gmail.com>
0 siblings, 0 replies; 182+ messages in thread
From: David G. Johnston @ 2021-03-08 22:37 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>; k.jamison@fujitsu.com, Craig Ringer <craig.ringer@enterprisedb.com>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
On Wed, Mar 3, 2021 at 5:45 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> I'm proposing some minor changes.
>
>
Some additional tweaks/comments for the documentation with the edit
proposed edits:
(edit) + <function>PQresultStatus</function>, will report a
Remove the comma
(orig) + the failed operation are skipped entirely. The same behaviour
holds if the
We seem to use "behavior" more frequently
(edit) + From the client perspective, after
<function>PQresultStatus</function>
Possessive "client's perspective"
(edit) + its expected results queue. Based on available memory,
results from the
"its corresponding results queue" - to go along with this change:
- of the order in which it sent queries and the expected results.
+ of the order in which it sent queries, to associate them with their
+ corresponding results.
(orig)
+ pipeline mode. If the current statement isn't finished processing
+ or there are results pending for collection with
Needs a comma after processing.
"results pending for collection" reads oddly to me...not sure what would be
better though...
(edit)
+ <note>
+ <para>
+ The pipeline API was introduced in
<productname>PostgreSQL</productname> 14.
+ Pipeline mode is a client-side feature which doesn't require server
+ support, and works on any server that supports the v3 extended query
+ protocol.
+ </para>
+ </note>
This note seems like it should be placed either near the very beginning of
the feature or incorporated into the feature introduction.
(orig)
+ If any statement encounters an error, the server aborts the current
+(-) transaction and skips processing commands in the pipeline until the
+ next synchronization point established by
<function>PQsendPipeline</function>.
I dislike "skips" here - even if it doesn't execute a command it still will
place a result on the socket so that the client can have something to match
up with the queries it sent, correct?
+ transaction and creates a PGRES_PIPELINE_ABORTED result for all commands
in the pipeline until the
David J.
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2021-01-22 00:14 ` Zhihong Yu <zyu@yugabyte.com>
2021-02-19 18:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
1 sibling, 1 reply; 182+ messages in thread
From: Zhihong Yu @ 2021-01-22 00:14 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: David G. Johnston <david.g.johnston@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael.paquier@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; Heikki Linnakangas <heikki.linnakangas@iki.fi>
Hi,
+ commandFailed(st, "SQL", "\\gset and \\aset are not
allowed in a batch section");
It seems '\\gset or \\aset is not ' would correspond to the check more
closely.
+ if (my_command->argc != 1)
+ syntax_error(source, lineno, my_command->first_line,
my_command->argv[0],
It is possible that my_command->argc == 0 (where my_command->argv[0]
shouldn't be accessed) ?
+ appendPQExpBufferStr(&conn->errorMessage,
+ libpq_gettext("cannot queue commands
during COPY\n"));
+ return false;
+ break;
Is the break necessary ? Similar comment for pqBatchProcessQueue().
+int
+PQexitBatchMode(PGconn *conn)
Since either 0 or 1 is returned, maybe change the return type to bool.
Also, the function operates on PGconn - should the function be static
(pqBatchProcessQueue is) ?
Cheers
On Thu, Jan 21, 2021 at 3:39 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:
> Thanks David Johnston and Daniel Vérité, I have incorporated your
> changes into this patch, which is now v26. Also, it's been rebased on
> current sources.
>
> I've been using the new PQtrace() stuff to verify the behavior of the
> new feature. It's not perfect, but at least it doesn't crash
> immediately as it did when I tried a few weeks ago. There are
> imperfections that I think are due to bugs in the PQtrace
> implementation, not in this patch.
>
> --
> Álvaro Herrera 39°49'30"S 73°17'W
> "El conflicto es el camino real hacia la unión"
>
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-22 00:14 ` Re: PATCH: Batch/pipelining support for libpq Zhihong Yu <zyu@yugabyte.com>
@ 2021-02-19 18:43 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-19 20:08 ` Re: PATCH: Batch/pipelining support for libpq Zhihong Yu <zyu@yugabyte.com>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-02-19 18:43 UTC (permalink / raw)
To: Zhihong Yu <zyu@yugabyte.com>; +Cc: David G. Johnston <david.g.johnston@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael.paquier@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; Heikki Linnakangas <heikki.linnakangas@iki.fi>
On 2021-Jan-21, Zhihong Yu wrote:
> It seems '\\gset or \\aset is not ' would correspond to the check more
> closely.
>
> + if (my_command->argc != 1)
> + syntax_error(source, lineno, my_command->first_line,
> my_command->argv[0],
>
> It is possible that my_command->argc == 0 (where my_command->argv[0]
> shouldn't be accessed) ?
No -- process_backslash_command reads the word and sets it as argv[0].
> + appendPQExpBufferStr(&conn->errorMessage,
> + libpq_gettext("cannot queue commands
> during COPY\n"));
> + return false;
> + break;
>
> Is the break necessary ? Similar comment for pqBatchProcessQueue().
Not necessary. I removed them.
> +int
> +PQexitBatchMode(PGconn *conn)
>
> Since either 0 or 1 is returned, maybe change the return type to bool.
I was kinda tempted to do that, but in reality a lot of libpq's API is
defined like that -- to return 0 (failure)/1 (success) as ints, not
bools. For example see PQsendQuery(). I'm not inclined to do different
for these new functions. (One curious case is PQsetvalue, which returns
int, and is documented as "returns zero for failure" and then uses
"return false").
> Also, the function operates on PGconn - should the function be static
> (pqBatchProcessQueue is) ?
I don't understand this suggestion. How would an application call it,
if we make it static?
Thanks
--
Álvaro Herrera Valdivia, Chile
"Cómo ponemos nuestros dedos en la arcilla del otro. Eso es la amistad; jugar
al alfarero y ver qué formas se pueden sacar del otro" (C. Halloway en
La Feria de las Tinieblas, R. Bradbury)
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-22 00:14 ` Re: PATCH: Batch/pipelining support for libpq Zhihong Yu <zyu@yugabyte.com>
2021-02-19 18:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2021-02-19 20:08 ` Zhihong Yu <zyu@yugabyte.com>
2021-02-19 20:46 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 182+ messages in thread
From: Zhihong Yu @ 2021-02-19 20:08 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: David G. Johnston <david.g.johnston@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael.paquier@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; Heikki Linnakangas <heikki.linnakangas@iki.fi>
Hi,
+static int pqBatchProcessQueue(PGconn *conn);
I was suggesting changing:
+int
+PQexitBatchMode(PGconn *conn)
to:
+static int
+PQexitBatchMode(PGconn *conn)
Cheers
On Fri, Feb 19, 2021 at 10:43 AM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:
> On 2021-Jan-21, Zhihong Yu wrote:
>
> > It seems '\\gset or \\aset is not ' would correspond to the check more
> > closely.
> >
> > + if (my_command->argc != 1)
> > + syntax_error(source, lineno, my_command->first_line,
> > my_command->argv[0],
> >
> > It is possible that my_command->argc == 0 (where my_command->argv[0]
> > shouldn't be accessed) ?
>
> No -- process_backslash_command reads the word and sets it as argv[0].
>
> > + appendPQExpBufferStr(&conn->errorMessage,
> > + libpq_gettext("cannot queue commands
> > during COPY\n"));
> > + return false;
> > + break;
> >
> > Is the break necessary ? Similar comment for pqBatchProcessQueue().
>
> Not necessary. I removed them.
>
> > +int
> > +PQexitBatchMode(PGconn *conn)
> >
> > Since either 0 or 1 is returned, maybe change the return type to bool.
>
> I was kinda tempted to do that, but in reality a lot of libpq's API is
> defined like that -- to return 0 (failure)/1 (success) as ints, not
> bools. For example see PQsendQuery(). I'm not inclined to do different
> for these new functions. (One curious case is PQsetvalue, which returns
> int, and is documented as "returns zero for failure" and then uses
> "return false").
>
> > Also, the function operates on PGconn - should the function be static
> > (pqBatchProcessQueue is) ?
>
> I don't understand this suggestion. How would an application call it,
> if we make it static?
>
> Thanks
>
> --
> Álvaro Herrera Valdivia, Chile
> "Cómo ponemos nuestros dedos en la arcilla del otro. Eso es la amistad;
> jugar
> al alfarero y ver qué formas se pueden sacar del otro" (C. Halloway en
> La Feria de las Tinieblas, R. Bradbury)
>
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-22 00:14 ` Re: PATCH: Batch/pipelining support for libpq Zhihong Yu <zyu@yugabyte.com>
2021-02-19 18:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-19 20:08 ` Re: PATCH: Batch/pipelining support for libpq Zhihong Yu <zyu@yugabyte.com>
@ 2021-02-19 20:46 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-19 21:08 ` Re: PATCH: Batch/pipelining support for libpq Zhihong Yu <zyu@yugabyte.com>
0 siblings, 1 reply; 182+ messages in thread
From: Alvaro Herrera @ 2021-02-19 20:46 UTC (permalink / raw)
To: Zhihong Yu <zyu@yugabyte.com>; +Cc: David G. Johnston <david.g.johnston@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael.paquier@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; Heikki Linnakangas <heikki.linnakangas@iki.fi>
Hi,
On 2021-Feb-19, Zhihong Yu wrote:
> Hi,
> +static int pqBatchProcessQueue(PGconn *conn);
>
> I was suggesting changing:
>
> +int
> +PQexitBatchMode(PGconn *conn)
>
> to:
>
> +static int
> +PQexitBatchMode(PGconn *conn)
I understand that, but what I'm saying is that it doesn't work.
pqBatchProcessQueue can be static because it's only used internally in
libpq, but PQexitBatchMode is supposed to be called by the client
application.
--
Álvaro Herrera Valdivia, Chile
"No necesitamos banderas
No reconocemos fronteras" (Jorge González)
^ permalink raw reply [nested|flat] 182+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:08 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Re: PATCH: Batch/pipelining support for libpq Dave Cramer <davecramer@postgres.rocks>
2020-09-21 17:55 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Re: PATCH: Batch/pipelining support for libpq Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 17:20 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 13:40 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` Re: PATCH: Batch/pipelining support for libpq David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-22 00:14 ` Re: PATCH: Batch/pipelining support for libpq Zhihong Yu <zyu@yugabyte.com>
2021-02-19 18:43 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-19 20:08 ` Re: PATCH: Batch/pipelining support for libpq Zhihong Yu <zyu@yugabyte.com>
2021-02-19 20:46 ` Re: PATCH: Batch/pipelining support for libpq Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2021-02-19 21:08 ` Zhihong Yu <zyu@yugabyte.com>
0 siblings, 0 replies; 182+ messages in thread
From: Zhihong Yu @ 2021-02-19 21:08 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: David G. Johnston <david.g.johnston@gmail.com>; Daniel Verite <daniel@manitou-mail.org>; Matthieu Garrigues <matthieu.garrigues@gmail.com>; Michael Paquier <michael@paquier.xyz>; Dave Cramer <davecramer@postgres.rocks>; pgsql-hackers; Craig Ringer <craig@2ndquadrant.com>; Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael.paquier@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; Heikki Linnakangas <heikki.linnakangas@iki.fi>
Hi,
Thanks for the response.
On Fri, Feb 19, 2021 at 12:46 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:
> Hi,
>
> On 2021-Feb-19, Zhihong Yu wrote:
>
> > Hi,
> > +static int pqBatchProcessQueue(PGconn *conn);
> >
> > I was suggesting changing:
> >
> > +int
> > +PQexitBatchMode(PGconn *conn)
> >
> > to:
> >
> > +static int
> > +PQexitBatchMode(PGconn *conn)
>
> I understand that, but what I'm saying is that it doesn't work.
> pqBatchProcessQueue can be static because it's only used internally in
> libpq, but PQexitBatchMode is supposed to be called by the client
> application.
>
> --
> Álvaro Herrera Valdivia, Chile
> "No necesitamos banderas
> No reconocemos fronteras" (Jorge González)
>
^ permalink raw reply [nested|flat] 182+ messages in thread
end of thread, other threads:[~2021-03-18 11:28 UTC | newest]
Thread overview: 182+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2016-05-23 09:19 PATCH: Batch/pipelining support for libpq Craig Ringer <craig@2ndquadrant.com>
2016-05-23 15:50 ` Andres Freund <andres@anarazel.de>
2016-05-23 16:00 ` Michael Paquier <michael.paquier@gmail.com>
2016-05-24 01:20 ` Craig Ringer <craig@2ndquadrant.com>
2016-05-24 02:01 ` Craig Ringer <craig@2ndquadrant.com>
2016-05-27 02:43 ` Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-06-03 11:51 ` Dmitry Igrishin <dmitigr@gmail.com>
2016-08-10 06:44 ` Michael Paquier <michael.paquier@gmail.com>
2016-08-23 00:27 ` Craig Ringer <craig@2ndquadrant.com>
2016-08-23 06:56 ` Craig Ringer <craig@2ndquadrant.com>
2016-09-06 08:10 ` Daniel Verite <daniel@manitou-mail.org>
2016-09-06 11:01 ` Craig Ringer <craig@2ndquadrant.com>
2016-10-03 02:10 ` Michael Paquier <michael.paquier@gmail.com>
2016-10-03 03:48 ` Craig Ringer <craig@2ndquadrant.com>
2016-10-03 14:52 ` Daniel Verite <daniel@manitou-mail.org>
2016-10-04 07:15 ` Michael Paquier <michael.paquier@gmail.com>
2016-10-04 08:38 ` Craig Ringer <craig@2ndquadrant.com>
2016-10-04 08:39 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
2016-10-05 04:54 ` Michael Paquier <michael.paquier@gmail.com>
2016-10-14 01:04 ` Jim Nasby <Jim.Nasby@BlueTreble.com>
2016-11-18 06:04 ` Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
2016-11-18 08:18 ` Craig Ringer <craig@2ndquadrant.com>
2016-11-22 07:14 ` Haribabu Kommi <kommi.haribabu@gmail.com>
2016-11-22 07:31 ` Craig Ringer <craig@2ndquadrant.com>
2017-02-14 07:27 ` Iwata, Aya <iwata.aya@jp.fujitsu.com>
2017-02-17 05:17 ` Prabakaran, Vaishnavi <VaishnaviP@fast.au.fujitsu.com>
2017-02-21 07:51 ` Michael Paquier <michael.paquier@gmail.com>
2017-02-22 06:14 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-02-23 02:37 ` Craig Ringer <craig@2ndquadrant.com>
2017-03-07 16:52 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-08 01:56 ` Craig Ringer <craig@2ndquadrant.com>
2017-03-08 02:04 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-09 21:43 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-10 13:52 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-13 00:54 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-13 01:32 ` Craig Ringer <craig@2ndquadrant.com>
2017-03-13 17:19 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-14 06:50 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 02:42 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-16 13:37 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-20 01:32 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-24 15:21 ` David Steele <david@pgmasters.net>
2017-03-25 12:50 ` Craig Ringer <craig@2ndquadrant.com>
2017-03-27 07:26 ` Michael Paquier <michael.paquier@gmail.com>
2017-03-27 07:42 ` Craig Ringer <craig@2ndquadrant.com>
2017-03-27 08:18 ` Michael Paquier <michael.paquier@gmail.com>
2017-03-27 13:28 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-27 15:27 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-28 04:57 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-28 08:10 ` Michael Paquier <michael.paquier@gmail.com>
2017-03-28 13:28 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-29 03:40 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 01:08 ` Michael Paquier <michael.paquier@gmail.com>
2017-03-30 05:02 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-03-30 18:12 ` Daniel Verite <daniel@manitou-mail.org>
2017-03-31 15:03 ` David Steele <david@pgmasters.net>
2017-04-03 04:10 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 21:05 ` Andres Freund <andres@anarazel.de>
2017-04-03 23:24 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-03 23:26 ` Andres Freund <andres@anarazel.de>
2017-04-03 23:57 ` Michael Paquier <michael.paquier@gmail.com>
2017-04-04 04:52 ` Andres Freund <andres@anarazel.de>
2017-04-05 07:00 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-04-05 22:45 ` Andres Freund <andres@anarazel.de>
2017-06-19 22:49 ` Andres Freund <andres@anarazel.de>
2017-06-20 00:09 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-06-20 01:43 ` Craig Ringer <craig@2ndquadrant.com>
2017-06-20 02:11 ` Michael Paquier <michael.paquier@gmail.com>
2017-06-20 15:51 ` Daniel Verite <daniel@manitou-mail.org>
2017-06-21 23:40 ` Andres Freund <andres@anarazel.de>
2017-06-21 23:44 ` Craig Ringer <craig@2ndquadrant.com>
2017-06-22 00:29 ` Andres Freund <andres@anarazel.de>
2017-06-22 01:03 ` Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:07 ` Andres Freund <andres@anarazel.de>
2017-06-22 01:18 ` Craig Ringer <craig@2ndquadrant.com>
2017-06-22 01:21 ` Andres Freund <andres@anarazel.de>
2017-06-22 11:43 ` Daniel Verite <daniel@manitou-mail.org>
2017-06-22 15:52 ` Andres Freund <andres@anarazel.de>
2017-06-22 17:42 ` Daniel Verite <daniel@manitou-mail.org>
2017-08-10 05:23 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-08-23 09:40 ` Andres Freund <andres@anarazel.de>
2017-09-13 05:06 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 05:33 ` Craig Ringer <craig@2ndquadrant.com>
2017-09-13 05:44 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-09-13 06:01 ` Craig Ringer <craig@2ndquadrant.com>
2017-10-02 09:31 ` Daniel Gustafsson <daniel@yesql.se>
2017-10-05 00:58 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2017-11-28 01:57 ` Michael Paquier <michael.paquier@gmail.com>
2018-01-05 05:55 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-01-11 23:12 ` Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
2018-03-22 12:11 ` Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
2018-03-23 13:18 ` Daniel Verite <daniel@manitou-mail.org>
2018-10-01 06:34 ` Michael Paquier <michael@paquier.xyz>
2018-11-29 16:14 ` Dmitry Dolgov <9erthalion6@gmail.com>
2019-08-30 13:36 ` Nikhil Sontakke <nikhils@2ndquadrant.com>
2019-09-09 07:20 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 23:01 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2020-07-14 19:18 ` Andres Freund <andres@anarazel.de>
2021-03-04 11:22 ` Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 14:26 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 15:01 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 16:41 ` Ibrar Ahmed <ibrar.ahmad@gmail.com>
2021-03-04 16:47 ` Justin Pryzby <pryzby@telsasoft.com>
2021-03-04 17:05 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:24 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:48 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 19:59 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 20:51 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-04 21:40 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-06 00:31 ` Zhihong Yu <zyu@yugabyte.com>
2021-03-06 00:35 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-11 16:00 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-11 18:49 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-03-13 04:28 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-15 18:54 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-15 19:00 ` Justin Pryzby <pryzby@telsasoft.com>
2021-03-15 19:47 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2021-03-16 20:00 ` Andres Freund <andres@anarazel.de>
2021-03-18 11:28 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
2017-06-22 11:23 ` Daniel Verite <daniel@manitou-mail.org>
2016-05-24 04:02 ` Craig Ringer <craig@2ndquadrant.com>
2016-05-24 04:28 ` Tom Lane <tgl@sss.pgh.pa.us>
2016-05-24 13:55 ` Jim Nasby <Jim.Nasby@BlueTreble.com>
2016-10-12 11:51 ` Shay Rojansky <roji@roji.org>
2016-10-14 05:02 ` Craig Ringer <craig@2ndquadrant.com>
2016-10-14 10:09 ` Shay Rojansky <roji@roji.org>
2016-10-14 12:53 ` Craig Ringer <craig@2ndquadrant.com>
2016-10-14 14:15 ` Shay Rojansky <roji@roji.org>
2016-10-14 14:28 ` Craig Ringer <craig@2ndquadrant.com>
2020-08-31 16:05 Re: PATCH: Batch/pipelining support for libpq Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 12:38 ` Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:21 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2020-09-21 13:08 ` Dave Cramer <davecramer@postgres.rocks>
2020-09-21 13:41 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 13:39 ` Dave Cramer <davecramer@postgres.rocks>
2020-09-21 14:22 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-09-21 17:55 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 04:35 ` Michael Paquier <michael@paquier.xyz>
2020-10-01 06:41 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-01 08:13 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-10-26 19:09 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:45 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-02 15:57 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 13:20 ` Dave Cramer <davecramer@postgres.rocks>
2020-11-03 13:42 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-03 15:56 ` Dave Cramer <davecramer@postgres.rocks>
2020-11-03 16:44 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-11-11 22:46 ` Andres Freund <andres@anarazel.de>
2020-11-03 17:20 ` David G. Johnston <david.g.johnston@gmail.com>
2020-11-12 09:46 ` Matthieu Garrigues <matthieu.garrigues@gmail.com>
2020-11-12 13:40 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 00:37 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-14 14:42 ` Daniel Verite <daniel@manitou-mail.org>
2020-11-18 17:51 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-18 23:17 ` Andres Freund <andres@anarazel.de>
2020-11-23 21:58 ` Daniel Verite <daniel@manitou-mail.org>
2020-11-23 22:15 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2020-11-16 23:29 ` David G. Johnston <david.g.johnston@gmail.com>
2021-01-21 23:39 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-21 23:43 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-10 23:51 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 01:19 ` Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 01:38 ` Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-16 16:27 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:13 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-16 23:45 ` Zhihong Yu <zyu@yugabyte.com>
2021-02-19 18:35 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-20 00:23 ` Craig Ringer <craig.ringer@enterprisedb.com>
2021-02-20 01:36 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-03-03 07:07 ` k.jamison@fujitsu.com <k.jamison@fujitsu.com>
2021-03-03 16:34 ` 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-03 23:42 ` 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-04 00:24 ` 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-04 01:51 ` 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-08 23:02 ` 'Alvaro Herrera' <alvherre@alvh.no-ip.org>
2021-03-04 00:45 ` Justin Pryzby <pryzby@telsasoft.com>
2021-03-08 22:37 ` David G. Johnston <david.g.johnston@gmail.com>
2021-01-22 00:14 ` Zhihong Yu <zyu@yugabyte.com>
2021-02-19 18:43 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-19 20:08 ` Zhihong Yu <zyu@yugabyte.com>
2021-02-19 20:46 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-02-19 21:08 ` Zhihong Yu <zyu@yugabyte.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox