Subject: Re: pg_upgrade failing for 200+ million Large Objects
To: "Tharakan, Robins" <tharar@amazon.com>,
 "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
References: <2543eda3e609455e9cf283646a8b2bc8@EX13D05UWC001.ant.amazon.com>
From: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Message-ID: <cc089cc3-fc43-9904-fdba-d830d8222145@enterprisedb.com>
Date: Mon, 8 Mar 2021 11:25:13 +0100
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0)
 Gecko/20100101 Thunderbird/78.8.0
MIME-Version: 1.0
In-Reply-To: <2543eda3e609455e9cf283646a8b2bc8@EX13D05UWC001.ant.amazon.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/cc089cc3-fc43-9904-fdba-d830d8222145%40enterprisedb.com>
Precedence: bulk

On 07.03.21 09:43, Tharakan, Robins wrote:
> Attached is a proof-of-concept patch that allows Postgres to perform
> pg_upgrade if the instance has Millions of objects.
> 
> It would be great if someone could take a look and see if this patch is in
> the right direction. There are some pending tasks (such as documentation /
> pg_resetxlog vs pg_resetwal related changes) but for now, the patch helps
> remove a stalemate where if a Postgres instance has a large number
> (accurately speaking 146+ Million) of Large Objects, pg_upgrade fails. This
> is easily reproducible and besides deleting Large Objects before upgrade,
> there is no other (apparent) way for pg_upgrade to complete.
> 
> The patch (attached):
> - Applies cleanly on REL9_6_STABLE -
> c7a4fc3dd001646d5938687ad59ab84545d5d043
> - 'make check' passes
> - Allows the user to provide a constant via pg_upgrade command-line, that
> overrides the 2 billion constant in pg_resetxlog [1] thereby increasing the
> (window of) Transaction IDs available for pg_upgrade to complete.

Could you explain what your analysis of the problem is and why this 
patch (might) fix it?

Right now, all I see here is, pass a big number via a command-line 
option and hope it works.