From: Tom Lane <tgl@sss.pgh.pa.us>
To: "Kumar, Sachin" <ssetiya@amazon.com>
cc: Robins Tharakan <tharakan@gmail.com>,
        Nathan Bossart <nathandbossart@gmail.com>, Jan Wieck <jan@wi3ck.info>,
        Bruce Momjian <bruce@momjian.us>, Zhihong Yu <zyu@yugabyte.com>,
        "Andrew Dunstan" <andrew@dunslane.net>,
        Magnus Hagander <magnus@hagander.net>,
        Peter Eisentraut <peter.eisentraut@enterprisedb.com>,
        "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: pg_upgrade failing for 200+ million Large Objects
In-reply-to: <557FD681-3929-44A1-87B2-6B5E10C4A66B@amazon.com>
References: <c0f5fde5-5cad-65ed-dd30-cd852301d074@wi3ck.info> <986904.1616525964@sss.pgh.pa.us> <6cccaa33-c263-b8a2-b064-985605d33d25@wi3ck.info> <988415.1616528159@sss.pgh.pa.us> <872315a8-99fc-da4e-463d-784cfb5a025d@wi3ck.info> <1010642.1616532950@sss.pgh.pa.us> <802b96e9-f5e1-015c-dfb9-8756974b11fc@wi3ck.info> <0263bf35-05d6-02a1-519b-b7895a918314@wi3ck.info> <20220825003227.GA1456581@nathanxps13> <663393ca-b2ff-26f0-2e2d-adc942aff4fd@timescale.com> <20220908231807.GA2242918@nathanxps13> <CAAWbhmgUb8p7ff_ZX5jCvqM=ipPxbbDJTXMNVzH-Ho_CXVkRHA@mail.gmail.com> <0643CC11-223A-4039-AC34-94E127462796@amazon.com> <1152134.1699555261@sss.pgh.pa.us> <83D44BE5-0088-4D41-8AE6-20A05D026F46@amazon.com> <81D13E16-BA04-43CF-9B89-B8924300B211@amazon.com> <240D05EC-8B28-4112-BEAB-85ECBAF3F871@amazon.com> <2055911.1702258962@sss.pgh.pa.us> <557FD681-3929-44A1-87B2-6B5E10C4A66B@amazon.com>
Comments: In-reply-to "Kumar, Sachin" <ssetiya@amazon.com>
	message dated "Tue, 02 Jan 2024 17:33:00 +0000"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <4107507.1704218778.1@sss.pgh.pa.us>
Content-Transfer-Encoding: quoted-printable
Date: Tue, 02 Jan 2024 13:06:18 -0500
Message-ID: <4107508.1704218778@sss.pgh.pa.us>
Archived-At: <https://www.postgresql.org/message-id/4107508.1704218778%40sss.pgh.pa.us>
Precedence: bulk

"Kumar, Sachin" <ssetiya@amazon.com> writes:
>> On 11/12/2023, 01:43, "Tom Lane" <tgl@sss.pgh.pa.us <mailto:tgl@sss.pgh=
.pa.us>> wrote:
>> ... Maybe that
>> could be improved in future, but it seems like it'd add a
>> lot more complexity, and it wouldn't make life any better for
>> pg_upgrade (which doesn't use parallel pg_restore, and seems
>> unlikely to want to in future).

> I was not able to find email thread which details why we are not using
> parallel pg_restore for pg_upgrade.

Well, it's pretty obvious isn't it?  The parallelism is being applied
at the per-database level instead.

> IMHO most of the customer will have single large
> database, and not using parallel restore will cause slow pg_upgrade.

You've offered no justification for that opinion ...

> I am attaching a patch which enables parallel pg_restore for DATA and PO=
ST-DATA part
> of dump. It will push down --jobs value to pg_restore and will restore
> database sequentially.

I don't think I trust this patch one bit.  It makes way too many
assumptions about how the --section options work, or even that they
will work at all in a binary-upgrade situation.  I've spent enough
time with that code to know that --section is pretty close to being
a fiction.  One point in particular is that this would change the
order of ACL restore relative to other steps, which almost certainly
will cause problems for somebody.

			regards, tom lane