Date: Fri, 18 Jun 2021 13:39:52 -0400
From: Alvaro Herrera <alvaro.herrera@2ndquadrant.com>
To: Boris Kolpackov <boris@codesynthesis.com>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: Pipeline mode and PQpipelineSync()
Message-ID: <202106181739.7krtfdwkjbic@alvherre.pgsql>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <boris.20210616110321@codesynthesis.com>
Archived-At: 
 <https://www.postgresql.org/message-id/202106181739.7krtfdwkjbic%40alvherre.pgsql>
Precedence: bulk

On 2021-Jun-16, Boris Kolpackov wrote:

> Specifically, the documentation[1]
> makes it sound like the use of PQpipelineSync() is optional (34.5.1.1
> "Issuing Queries"):

Hmm.  My intention here was to indicate that you should have
PQpipelineSync *somewhere*, but that the server was free to start
executing some commands even before that, if the buffered commands
happened to reach the server somehow -- but not necessarily that the
results from those commands would reach the client immediately.

I'll experiment a bit more to be sure that what I'm saying is correct.
But if it is, then I think the documentation you quote is misleading:

> "After entering pipeline mode, the application dispatches requests using
> PQsendQuery, PQsendQueryParams, or its prepared-query sibling
> PQsendQueryPrepared. These requests are queued on the client-side until
> flushed to the server; this occurs when PQpipelineSync is used to establish a
> synchronization point in the pipeline, or when PQflush is called. [...]
> 
> The server executes statements, and returns results, in the order the client
> sends them. The server will begin executing the commands in the pipeline
> immediately, not waiting for the end of the pipeline. [...]"

... because it'll lead people to do what you've done, only to discover
that it doesn't really work.

I think I should rephrase this to say that PQpipelineSync() is needed
where the user needs the server to start executing commands; and
separately indicate that it is possible (but not promised) that the
server would start executing commands ahead of time because $reasons.

Do I have it right that other than this documentation problem, you've
been able to use pipeline mode successfully?

> So to me it looks like, contrary to the documentation, the server does
> not start executing the statements immediately, instead waiting for the
> synchronization point. Or am I missing something here?

I don't think you are.

> The above tests were performed using libpq from 14beta1 running against
> PostgreSQL server version 9.5. If you would like to take a look at the
> actual code, you can find it here[2] (the PIPELINE_SYNC macro controls
> whether PQpipelineSync() is used).

Thanks.

> On a related note, I've been using libpq_pipeline.c[3] as a reference
> and I believe it has a busy loop calling PQflush() repeatedly on line
> 721 since once everything has been sent and we are waiting for the
> result, select() will keep returning with an indication that the socket
> is writable

Oops, thanks, will look at fixing this too.

> (you can find one way to fix this in [2]).
> [2] https://git.codesynthesis.com/cgit/odb/libodb-pgsql/tree/odb/pgsql/statement.cxx?h=bulk#n771

Neat, can do likewise I suppose.

-- 
Álvaro Herrera                            39°49'30"S 73°17'W