Date: Fri, 26 Aug 2022 15:04:46 -0500
From: Justin Pryzby <pryzby@telsasoft.com>
To: Bruce Momjian <bruce@momjian.us>
Cc: "Finnerty, Jim" <jfinnert@amazon.com>,
	Stephen Frost <sfrost@snowman.net>, Maxim Orlov <orlovmg@gmail.com>,
	pgsql-hackers@postgresql.org, Magnus Hagander <magnus@hagander.net>,
	Michael Paquier <michael@paquier.xyz>
Subject: Re: Add 64-bit XIDs into PostgreSQL 15
Message-ID: <20220826200446.GK2342@telsasoft.com>
References: 
 <CACG=ezZe1NQSCnfHOr78AtAZxJZeCvxrts0ygrxYwe=pyyjVWA@mail.gmail.com>
 <20220104193220.GL15820@tamriel.snowman.net>
 <CB919F14-2B98-4CD3-84C0-91945E9E7EF5@amazon.com>
 <YdYvCT3LmItnTTwz@momjian.us>
 <20220106001226.GS14051@telsasoft.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220106001226.GS14051@telsasoft.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Archived-At: 
 <https://www.postgresql.org/message-id/20220826200446.GK2342%40telsasoft.com>
Precedence: bulk

On Wed, Jan 05, 2022 at 06:12:26PM -0600, Justin Pryzby wrote:
> On Wed, Jan 05, 2022 at 06:51:37PM -0500, Bruce Momjian wrote:
> > On Tue, Jan 4, 2022 at 10:22:50PM +0000, Finnerty, Jim wrote:
> > > I'm concerned about the maintainability impact of having 2 new
> > > on-disk page formats.  It's already complex enough with XIDs and
> > > multixact-XIDs.
> > >
> > > If the lack of space for the two epochs in the special data area is
> > > a problem only in an upgrade scenario, why not resolve the problem
> > > before completing the upgrade process like a kind of post-process
> > > pg_repack operation that converts all "double xmax" pages to
> > > the "double-epoch" page format? i.e. maybe the "double xmax"
> > > representation is needed as an intermediate representation during
> > > upgrade, but after upgrade completes successfully there are no pages
> > > with the "double-xmax" representation.  This would eliminate a whole
> > > class of coding errors and would make the code dealing with 64-bit
> > > XIDs simpler and more maintainable.
> > 
> > Well, yes, we could do this, and it would avoid the complexity of having
> > to support two XID representations, but we would need to accept that
> > fast pg_upgrade would be impossible in such cases, since every page
> > would need to be checked and potentially updated.
> > 
> > You might try to do this while the server is first started and running
> > queries, but I think we found out from the online checkpoint patch that
> 
> I think you meant the online checksum patch.  Which this reminded me of, too.

I wondered whether anyone had considered using relation forks to maintain state
of these long, transitional processes.

Either a whole new fork, or additional bits in the visibility map, which has
page-level bits.

There'd still need to be a flag somewhere indicating whether
checksums/xid64s/etc were enabled cluster-wide.  The VM/fork bits would need to
be checked while the cluster was being re-processed online.  This would add
some overhead.  After the cluster had reached its target state, the flag could
be set, and the VM bits would no longer need to be checked.

-- 
Justin