Message-ID: <6603e4e0.500a0220.a557f.4f39@mx.google.com>
Sender: Michael Banck <mbanck@gmail.com>
Date: Wed, 27 Mar 2024 10:20:31 +0100
From: Michael Banck <mbanck@gmx.net>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Laurenz Albe <laurenz.albe@cybertec.at>,
	Nathan Bossart <nathandbossart@gmail.com>,
	vignesh C <vignesh21@gmail.com>,
	"Kumar, Sachin" <ssetiya@amazon.com>,
	Robins Tharakan <tharakan@gmail.com>, Jan Wieck <jan@wi3ck.info>,
	Bruce Momjian <bruce@momjian.us>,
	Andrew Dunstan <andrew@dunslane.net>,
	Magnus Hagander <magnus@hagander.net>,
	Peter Eisentraut <peter.eisentraut@enterprisedb.com>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: pg_upgrade failing for 200+ million Large Objects
References: <83D44BE5-0088-4D41-8AE6-20A05D026F46@amazon.com>
 <81D13E16-BA04-43CF-9B89-B8924300B211@amazon.com>
 <240D05EC-8B28-4112-BEAB-85ECBAF3F871@amazon.com>
 <2055911.1702258962@sss.pgh.pa.us>
 <557FD681-3929-44A1-87B2-6B5E10C4A66B@amazon.com>
 <CALDaNm08DLqkr6LQfB=AJ-wEOGtASW+rYGsD6Q56-bYBYoHFdA@mail.gmail.com>
 <842242.1706287466@sss.pgh.pa.us>
 <!2751089.1710544721@sss.pgh.pa.us>
 <4a3ebf7d81bfc6dd4d545e5b27d6e8f6c32d8937.camel@cybertec.at>
 <3023817.1710629175@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <3023817.1710629175@sss.pgh.pa.us>
User-Agent: Mutt/1.10.1 (2018-07-13)
Archived-At: <https://www.postgresql.org/message-id/6603e4e0.500a0220.a557f.4f39%40mx.google.com>
Precedence: bulk

Hi,

On Sat, Mar 16, 2024 at 06:46:15PM -0400, Tom Lane wrote:
> Laurenz Albe <laurenz.albe@cybertec.at> writes:
> > On Fri, 2024-03-15 at 19:18 -0400, Tom Lane wrote:
> >> This patch seems to have stalled out again.  In hopes of getting it
> >> over the finish line, I've done a bit more work to address the two
> >> loose ends I felt were probably essential to deal with:
> 
> > Applies and builds fine.
> > I didn't scrutinize the code, but I gave it a spin on a database with
> > 15 million (small) large objects.  I tried pg_upgrade --link with and
> > without the patch on a debug build with the default configuration.
> 
> Thanks for looking at it!
> 
> > Without the patch:
> > Runtime: 74.5 minutes
> 
> > With the patch:
> > Runtime: 70 minutes
> 
> Hm, I'd have hoped for a bit more runtime improvement.  

I also think that this is quite a large runtime for pg_upgrade, but the
more important savings should be the memory usage.

> But perhaps not --- most of the win we saw upthread was from
> parallelism, and I don't think you'd get any parallelism in a
> pg_upgrade with all the data in one database.  (Perhaps there is more
> to do there later, but I'm still not clear on how this should interact
> with the existing cross-DB parallelism; so I'm content to leave that
> question for another patch.)

What is the status of this? In the commitfest, this patch is marked as
"Needs Review" with Nathan as reviewer - Nathan, were you going to take
another look at this or was your mail from January 12th a full review?

My feeling is that this patch is "Ready for Committer" and it is Tom's
call to commit it during the next days or not.

I am +1 that this is an important feature/bug fix to have. Because we
have customers stuck on older versions due to their pathological large
objects usage, I did some benchmarks (jsut doing pg_dump, not
pg_upgarde) a while ago which were also very promising; however, I lost
the exact numbers/results. I am happy to do further tests if that is
required for this patch to go forward.

Also, is there a chance this is going to be back-patched? I guess it
would be enough if the ugprade target is v17 so it is less of a concern,
but it would be nice if people with millions of large objects are not
stuck until they are ready to ugprade to v17.


Michael