MIME-Version: 1.0
References: <c0f5fde5-5cad-65ed-dd30-cd852301d074@wi3ck.info>
 <986904.1616525964@sss.pgh.pa.us>
 <6cccaa33-c263-b8a2-b064-985605d33d25@wi3ck.info>
 <988415.1616528159@sss.pgh.pa.us>
 <872315a8-99fc-da4e-463d-784cfb5a025d@wi3ck.info>
 <1010642.1616532950@sss.pgh.pa.us>
 <802b96e9-f5e1-015c-dfb9-8756974b11fc@wi3ck.info>
 <0263bf35-05d6-02a1-519b-b7895a918314@wi3ck.info>
 <20220825003227.GA1456581@nathanxps13>
 <663393ca-b2ff-26f0-2e2d-adc942aff4fd@timescale.com>
 <20220908231807.GA2242918@nathanxps13>
 <CAAWbhmgUb8p7ff_ZX5jCvqM=ipPxbbDJTXMNVzH-Ho_CXVkRHA@mail.gmail.com>
 <0643CC11-223A-4039-AC34-94E127462796@amazon.com>
 <1152134.1699555261@sss.pgh.pa.us>
 <83D44BE5-0088-4D41-8AE6-20A05D026F46@amazon.com>
 <81D13E16-BA04-43CF-9B89-B8924300B211@amazon.com>
 <240D05EC-8B28-4112-BEAB-85ECBAF3F871@amazon.com>
 <2055911.1702258962@sss.pgh.pa.us>
 <557FD681-3929-44A1-87B2-6B5E10C4A66B@amazon.com>
In-Reply-To: <557FD681-3929-44A1-87B2-6B5E10C4A66B@amazon.com>
From: vignesh C <vignesh21@gmail.com>
Date: Fri, 26 Jan 2024 20:12:44 +0530
Message-ID: 
 <CALDaNm08DLqkr6LQfB=AJ-wEOGtASW+rYGsD6Q56-bYBYoHFdA@mail.gmail.com>
Subject: Re: pg_upgrade failing for 200+ million Large Objects
To: "Kumar, Sachin" <ssetiya@amazon.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Robins Tharakan <tharakan@gmail.com>,
	Nathan Bossart <nathandbossart@gmail.com>, Jan Wieck <jan@wi3ck.info>,
	Bruce Momjian <bruce@momjian.us>, Zhihong Yu <zyu@yugabyte.com>,
	Andrew Dunstan <andrew@dunslane.net>, Magnus Hagander <magnus@hagander.net>,
	Peter Eisentraut <peter.eisentraut@enterprisedb.com>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Archived-At: 
 <https://www.postgresql.org/message-id/CALDaNm08DLqkr6LQfB%3DAJ-wEOGtASW%2BrYGsD6Q56-bYBYoHFdA%40mail.gmail.com>
Precedence: bulk

On Tue, 2 Jan 2024 at 23:03, Kumar, Sachin <ssetiya@amazon.com> wrote:
>
> > On 11/12/2023, 01:43, "Tom Lane" <tgl@sss.pgh.pa.us <mailto:tgl@sss.pgh.pa.us>> wrote:
>
> > I had initially supposed that in a parallel restore we could
> > have child workers also commit after every N TOC items, but was
> > soon disabused of that idea. After a worker processes a TOC
> > item, any dependent items (such as index builds) might get
> > dispatched to some other worker, which had better be able to
> > see the results of the first worker's step. So at least in
> > this implementation, we disable the multi-command-per-COMMIT
> > behavior during the parallel part of the restore. Maybe that
> > could be improved in future, but it seems like it'd add a
> > lot more complexity, and it wouldn't make life any better for
> > pg_upgrade (which doesn't use parallel pg_restore, and seems
> > unlikely to want to in future).
>
> I was not able to find email thread which details why we are not using
> parallel pg_restore for pg_upgrade. IMHO most of the customer will have single large
> database, and not using parallel restore will cause slow pg_upgrade.
>
> I am attaching a patch which enables parallel pg_restore for DATA and POST-DATA part
> of dump. It will push down --jobs value to pg_restore and will restore database sequentially.

CFBot shows that the patch does not apply anymore as in [1]:
=== Applying patches on top of PostgreSQL commit ID
46a0cd4cefb4d9b462d8cc4df5e7ecdd190bea92 ===
=== applying patch ./v9-005-parallel_pg_restore.patch
patching file src/bin/pg_upgrade/pg_upgrade.c
Hunk #3 FAILED at 650.
1 out of 3 hunks FAILED -- saving rejects to file
src/bin/pg_upgrade/pg_upgrade.c.rej

Please post an updated version for the same.

[1] - http://cfbot.cputube.org/patch_46_4713.log

Regards,
Vignesh