Date: Thu, 8 Sep 2022 16:18:07 -0700
From: Nathan Bossart <nathandbossart@gmail.com>
To: Jacob Champion <jchampion@timescale.com>
Cc: Jan Wieck <jan@wi3ck.info>, Tom Lane <tgl@sss.pgh.pa.us>,
	Bruce Momjian <bruce@momjian.us>, Zhihong Yu <zyu@yugabyte.com>,
	Andrew Dunstan <andrew@dunslane.net>,
	Magnus Hagander <magnus@hagander.net>,
	Robins Tharakan <tharakan@gmail.com>,
	Peter Eisentraut <peter.eisentraut@enterprisedb.com>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: pg_upgrade failing for 200+ million Large Objects
Message-ID: <20220908231807.GA2242918@nathanxps13>
References: <c0f5fde5-5cad-65ed-dd30-cd852301d074@wi3ck.info>
 <986904.1616525964@sss.pgh.pa.us>
 <6cccaa33-c263-b8a2-b064-985605d33d25@wi3ck.info>
 <988415.1616528159@sss.pgh.pa.us>
 <872315a8-99fc-da4e-463d-784cfb5a025d@wi3ck.info>
 <1010642.1616532950@sss.pgh.pa.us>
 <802b96e9-f5e1-015c-dfb9-8756974b11fc@wi3ck.info>
 <0263bf35-05d6-02a1-519b-b7895a918314@wi3ck.info>
 <20220825003227.GA1456581@nathanxps13>
 <663393ca-b2ff-26f0-2e2d-adc942aff4fd@timescale.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <663393ca-b2ff-26f0-2e2d-adc942aff4fd@timescale.com>
Archived-At: 
 <https://www.postgresql.org/message-id/20220908231807.GA2242918%40nathanxps13>
Precedence: bulk

On Wed, Sep 07, 2022 at 02:42:05PM -0700, Jacob Champion wrote:
> Just to clarify, was Justin's statement upthread (that the XID problem
> is fixed) correct? And is this patch just trying to improve the
> remaining memory and lock usage problems?

I think "fixed" might not be totally accurate, but that is the gist.

> I took a quick look at the pg_upgrade diffs. I agree with Jan that the
> escaping problem is a pretty bad smell, but even putting that aside for
> a bit, is it safe to expose arbitrary options to pg_dump/restore during
> upgrade? It's super flexible, but I can imagine that some of those flags
> might really mess up the new cluster...
> 
> And yeah, if you choose to do that then you get to keep both pieces, I
> guess, but I like that pg_upgrade tries to be (IMO) fairly bulletproof.

IIUC the main benefit of this approach is that it isn't dependent on
binary-upgrade mode, which seems to be a goal based on the discussion
upthread [0].  I think it'd be easily possible to fix only pg_upgrade by
simply dumping and restoring pg_largeobject_metadata, as Andres suggested
in 2018 [1].  In fact, it seems like it ought to be possible to just copy
pg_largeobject_metadata's files as was done before 12a53c7.  AFAICT this
would only work for clusters upgrading from v12 and newer, and it'd break
if any of the underlying data types change their storage format.  This
seems unlikely for OIDs, but there is ongoing discussion about changing
aclitem.

I still think this is a problem worth fixing, but it's not yet clear how to
proceed.

[0] https://postgr.es/m/227228.1616259220%40sss.pgh.pa.us
[1] https://postgr.es/m/20181122001415.ef5bncxqin2y3esb%40alap3.anarazel.de

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com