Date: Sat, 10 Aug 2024 10:17:27 -0500
From: Nathan Bossart <nathandbossart@gmail.com>
To: Corey Huinker <corey.huinker@gmail.com>
Cc: Ilya Gladyshev <ilya.v.gladyshev@gmail.com>,
	Daniel Gustafsson <daniel@yesql.se>,
	Robert Haas <robertmhaas@gmail.com>, Jeff Davis <pgsql@j-davis.com>,
	PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Subject: Re: optimizing pg_upgrade's once-in-each-database steps
Message-ID: <ZreEh54Xr6D3Qy8k@nathan>
References: <ZprY4e23vR95Nc51@nathan>
 <Zp677slGLwhPhVWy@nathan>
 <10c1c8dd-4685-46d4-80be-56bdfca8659a@gmail.com>
 <ZqvJg_lTTTt82BfQ@nathan>
 <ZqwA_qngQM25FrjK@nathan>
 <60cbc507-4c36-4037-9af7-028b1c8455b5@gmail.com>
 <ZrJ3bu4og_Uj_9qj@nathan>
 <CADkLM=e0KxKJH0XSG6x2oGK7-W2Eet4AwwCqz5pxAG7QthX4Xw@mail.gmail.com>
 <ZrYrL8-LQ2LhSPQ4@nathan>
 <CADkLM=fc-wujxNtGpLSVrBAtWQgsW-8RYrMxVyQ1buJfQX9CHw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CADkLM=fc-wujxNtGpLSVrBAtWQgsW-8RYrMxVyQ1buJfQX9CHw@mail.gmail.com>
Archived-At: <https://www.postgresql.org/message-id/ZreEh54Xr6D3Qy8k%40nathan>
Precedence: bulk

On Fri, Aug 09, 2024 at 04:06:16PM -0400, Corey Huinker wrote:
>> I'll admit I hadn't really considered pipelining, but I'm tempted to say
>> that it's probably not worth the complexity.  Not only do most of the tasks
>> have only one step, but even tasks like the data types check are unlikely
>> to require more than a few queries for upgrades from supported versions.
> 
> Can you point me to a complex multi-step task that you think wouldn't work
> for pipelining? My skimming of the other patches all seemed to be one query
> with one result set to be processed by one callback.

I think it would work fine.  I'm just not sure it's worth it, especially
for tasks that run one exactly one query in each connection.

>> Furthermore, most of the callbacks should do almost nothing for a given
>> upgrade, and since pg_upgrade runs on the server, client/server round-trip
>> time should be pretty low.
> 
> To my mind, that makes pipelining make more sense, you throw out N queries,
> most of which are trivial, and by the time you cycle back around and start
> digesting result sets via callbacks, more of the queries have finished
> because they were waiting on the query ahead of them in the pipeline, not
> waiting on a callback to finish consuming its assigned result set and then
> launching the next task query.

My assumption is that the "waiting for a callback before launching the next
query" time will typically be pretty short in practice.  I could try
measuring it...

>> Perhaps pipelining would make more sense if we consolidated the tasks a bit
>> better, but when I last looked into that, I didn't see a ton of great
>> opportunities that would help anything except for upgrades from really old
>> versions.  Even then, I'm not sure if pipelining is worth it.
> 
> I think you'd want to do the opposite of consolidating the tasks. If
> anything, you'd want to break them down in known single-query operations,
> and if the callback function for one of them happens to queue up a
> subsequent query (with subsequent callback) then so be it.

By "consolidating," I mean combining tasks into fewer tasks with additional
steps.  This would allow us to reuse connections instead of creating N
connections for every single query.  If we used a task per query, I'd
expect pipelining to provide zero benefit.

-- 
nathan