Message-ID: <1e2256b744836aeb485c61954e9d8272f80141a0.camel@j-davis.com>
Subject: Re: Comments on Custom RMGRs
From: Jeff Davis <pgsql@j-davis.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Danil Anisimow <anisimow.d@gmail.com>, Andres Freund
 <andres@anarazel.de>,  PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Date: Fri, 17 May 2024 13:20:19 -0700
In-Reply-To: <CA+TgmoZxvVXx809WhOcLNbUgAYYiLY+Y-o_gDpay6uTLwaJVyA@mail.gmail.com>
References: 
	<CANbhV-E4pTWeF-DsdaGsOrjJNFWPaR+Dstjrnkqvf9JFFgOKKQ@mail.gmail.com>
	 <f781f8f198567cee9e3776487bab3e41a7b91b82.camel@j-davis.com>
	 <20220512034010.4oqa76pasrulkw32@alap3.anarazel.de>
	 <CANbhV-G7SfdRLR6r7sxY0uL=TRU0S=cxkWJGWyTdss3u8LFK=g@mail.gmail.com>
	 <20220512234207.pwwp6q33f72byet2@alap3.anarazel.de>
	 <CANbhV-FFgAFeNZ=qNc7Fr991Asym+sfj=R5hb02XeRE6067=hg@mail.gmail.com>
	 <CABm2Ma7v4=W-1vH=kiA4M352F=Nd-Nvd_QBZufJ6EMGsS7Q-_A@mail.gmail.com>
	 <0892cd00635c8bcd458de6d43d31cf61953da1b2.camel@j-davis.com>
	 <CABm2Ma5Hi9PU_wWP0f=YFu9aDDdeSdX2+JFDL_oDH-QHgexsgg@mail.gmail.com>
	 <727b0f3b48aec2a4f968bf11c6fa8ca6382b6cca.camel@j-davis.com>
	 <CABm2Ma6avU5hkk7dd+G2oRNJZJtLOT=Yzeq6fb0v2rf7_YYp7g@mail.gmail.com>
	 <beb9c43d6492ffc273d236e4b93270dfad035f2f.camel@j-davis.com>
	 <CABm2Ma5Ww9M8DgFTCcASnMiPPVYU8cd_wGhHZ5YnoK5im6wT2g@mail.gmail.com>
	 <22e756affaad88b77a52d67cb532ed2a544f2e36.camel@j-davis.com>
	 <CA+TgmoZxvVXx809WhOcLNbUgAYYiLY+Y-o_gDpay6uTLwaJVyA@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.44.4-0ubuntu2 
MIME-Version: 1.0
Archived-At: <https://www.postgresql.org/message-id/1e2256b744836aeb485c61954e9d8272f80141a0.camel%40j-davis.com>
Precedence: bulk

On Fri, 2024-05-17 at 14:56 -0400, Robert Haas wrote:
> (2) a detailed
> description of how a non-core table AM or index AM is expected to be
> able to make use of this. Bonus points if the person providing that
> rationale can say credibly that they've actually implemented this and
> it works great with 100TB of production data.

That's a chicken-and-egg problem and we should be careful about setting
the bar too high for table AM improvements. Ultimately, AM authors will
benefit more from a steady stream of improvements that sometimes miss
the mark than complete stagnation, as long as we use reasonable
judgement.

There aren't a lot of table AMs, and to create a good one you need a
lot of internals knowledge. If it's an important AM, the developers are
surely going to try it out on mainline occasionally, and expect API
breaks. If the API breaks for them in some fundamental way, they can
complain and we still have time to fix it.

> The problem here is not only that we don't want to commit a hook that
> does nothing useful. We also don't want to commit a hook that works
> wonderfully for someone but we have no idea why. If we do that, then
> we don't know whether it's OK to modify the hook in the future as the
> code evolves, or more to the point, which kinds of modifications will
> be acceptable.

We have to have some kind of understanding between us and AM authors
that they need to participate in discussions when using these APIs, try
changes during development, be adaptable when they change from release
to release, and come back and tell us when something is wrong.

> And also, the next person who wants to use it is likely
> to have to figure out all on their own how to do so, instead of being
> able to refer to comments or documentation or the commit message or
> at
> least a mailing list post.

Obviously it would be better to have a nice example table AM in
/contrib, different enough from heap, but nobody has done that yet. You
could argue that we never should have exposed the API without something
like this in the first place, but that's also a big ask and we'd
probably still not have it.


Regarding this particular change: the checkpointing hook seems more
like a table AM feature, so I agree with you that we should have a good
idea how a real table AM might use this, rather than only
pg_stat_statements.

Regards,
	Jeff Davis