MIME-Version: 1.0
References: 
 <CA+TgmoZ-Jh1T6QyWoCODMVQdhTUPYkaZjWztzP1En4=ZHoKPzw@mail.gmail.com>
 <CAKZiRmxtJAFG7e1+Vs9B8ngON=AOzJbuws+1ZeH4LsbJh5AzoQ@mail.gmail.com>
 <CA+TgmoY9Ne_Sh10u6LSPc3wvOQPLp3kF9nBp3nqJEG2JGF2QiA@mail.gmail.com>
 <CA+Tgmoa57S6mP=aTOXH2-gDAL4TMO1WbGgrHSg0s6J4zUH=04g@mail.gmail.com>
 <DEB1SAGVDX7S.JMV96X4R0RCS@gmail.com>
 <CA+Tgmoaf__2B0BUL+vrg28P+3buX=Ti-kybqkHiLTtFrrCfzuA@mail.gmail.com>
 <CA+TgmoYpcLNOuypOTdgCSLW7FuA=t6BtB3meTARHX2-Dj_81xQ@mail.gmail.com>
 <DEFNP3WX7BLX.1X7DS1P9X6JIF@nmfay.com>
 <CA+TgmoZjv9OyFu1Gkt78w0vWEti8S33w8trYHmErf-GMmGSi=w@mail.gmail.com>
 <DELPC4B56Y5A.3OBZ3WCH3T4VX@nmfay.com>
 <CA+TgmoaOSBQD9Ux4eG40w723ZN=c0J7p-+oX4+J8urUeyLMo5w@mail.gmail.com>
 <CAOYmi+=g+MMoOpWkk2weXWKJcKH0eKey8gKHHdH0dF4Tiawrhw@mail.gmail.com>
 <CA+TgmobwaT=PXPDDrgDup+jA8KHBbkxigtziD-zNzAKKkQYVgQ@mail.gmail.com>
 <CAOYmi+mOmEW=amDRQMfw6-Fb3ZmDEQFaJzwk8Bc8W8DzaP85XQ@mail.gmail.com>
 <CA+TgmoaX2AMW4cdFM3OngBJxmxpkdmzF33R7-CWhvRLfucbFMg@mail.gmail.com>
 <CA+TgmoZ0x3ym_oueXRWzbM_=6ucKoPZVGj3rRMLBDC_FnetXDw@mail.gmail.com>
 <CAP53Pkycc=7N2bLzVT3x+qE1JamvRZWev5tFjdLJ1+-AV3Di+Q@mail.gmail.com>
 <CA+TgmoaKhuD91RnazbRyGkmP7--JdNq8oNDC3UcgTZSWbMxC7w@mail.gmail.com>
 <CAP53Pkw5-wMEeDJXFmqo_RTyL_spzCXb7HHKrbSnQqokVoZcNQ@mail.gmail.com>
 <CA+Tgmob-69bzbdi3U_QtebqAf6u1y8js=5=oNK639csVe1VbhA@mail.gmail.com>
 <CA+TgmoaZMOikxK=LqS+Jn+835h9S139JLGk-3LyETVXw5W5j=w@mail.gmail.com>
 <1136161.1769654478@sss.pgh.pa.us>
 <CAP53PkwZ1ZTMARKg6iEfAw9qzBhkjBitj-9gr_Jvy7k2AwGgWA@mail.gmail.com>
 <CAMbWs4--NuEUFE_xTo991TRXaZryE29jarJPDnVxoaQOYdt7tA@mail.gmail.com>
 <CA+TgmobzR+XMGbRosVPbjHbSo4+cgJn=qZK6w05aF1sbj=C+9Q@mail.gmail.com>
 <CA+TgmoawzvCoZAwFS85tE5+c8vBkqgcS8ZstQ_ohjXQ9wGT9sw@mail.gmail.com>
 <CA+TgmoYS4ZCVAF2jTce=bMP0Oq_db_srocR4cZyO0OBp9oUoGg@mail.gmail.com>
 <CAK98qZ2RzbgCHrSg4zLkvpzyBam_X6te-KF8w1+_vON9BAVMEw@mail.gmail.com>
 <CA+TgmoaCdsuvNn6T6SfQ_0YD2Hh2+hgTXh9fTGHQhPg1zvy2rQ@mail.gmail.com>
 <CA+Tgmob7ozJAs5SU6bD2RfAt4w_AmsMGz-aaVA6WeLXHkBypOg@mail.gmail.com>
 <CAK98qZ1J42RoAsHnYWGPPmHziZmzmqE=Lp_O6WJ-9aKK2qjikA@mail.gmail.com>
 <CA+TgmoYjcBA6dw3nwiyfDzPXTCrxTZPXDMrc2TrDJcL1cPK6iA@mail.gmail.com>
 <CA+TgmoYru-vxoTKfwjQby30r2OkTXfb18Km_=VLs6qk8Akr0-g@mail.gmail.com>
 <CA+Tgmoau7yJtvbeH-0kPt1Q=Gt_ezRdgM35Q1=LT665U_86Etg@mail.gmail.com>
 <1299934.1773938807@sss.pgh.pa.us>
 <CA+TgmobOLrMn5jEinWNPL5SrDH1DPpo3a4j+S6-4yhsZwWgzLg@mail.gmail.com>
 <CA+TgmoZUN8FT1Ah=m6Uis5bHa4FUa+_hMDWtcABG17toEfpiUg@mail.gmail.com>
 <CA+TgmoYh2-kM+tscOz=jVYq9Tf4SRPVqzPojs3KLZcW6E9m1BQ@mail.gmail.com>
 <CA+TgmoaK=4w7-qknUo3QhUJ53pXZq=c=KgZmRyD+k7ytqfmgSg@mail.gmail.com>
 <CAP53Pkz3DSFaaowYvbO5LULf3NhydD_UhHkighfWf6_pwxiqUw@mail.gmail.com>
 <CA+TgmoZ45n5jaNKKgbbj4-kYV8WsPvUn=Z8HnoZ7tUb_p9WKXg@mail.gmail.com>
 <CA+TgmoYuWmN-00Ec5pY7zAcpSFQUQLbgAdVWGR9kOR-HM-fHrA@mail.gmail.com>
 <CAKZiRmwFKhVz-HWvQmgPY7nZES9mCqdXHD++m9b7F4digcJpRQ@mail.gmail.com>
 <CAP53Pkxak127z_Xp7xW5G0+X4FELCBzNyoWCqTQ-xGfLog3KyQ@mail.gmail.com>
 <CA+Tgmoandcm1p=BCwLZPGRG_cno3pKSj4Ejt8AQZ4Hu-yhP1Mw@mail.gmail.com>
 <CAP53PkxJp=9PMzD8mKB3=4fK3Qp1iFZnA4pQSavP0nkpiNyoaA@mail.gmail.com>
In-Reply-To: 
 <CAP53PkxJp=9PMzD8mKB3=4fK3Qp1iFZnA4pQSavP0nkpiNyoaA@mail.gmail.com>
From: Robert Haas <robertmhaas@gmail.com>
Date: Mon, 30 Mar 2026 10:53:30 -0400
Message-ID: 
 <CA+TgmobnwuyTgYXweQrvOz_kFuBuCbKhsLz-mUPasFw_DQ8f6w@mail.gmail.com>
Subject: Re: pg_plan_advice
To: Lukas Fittl <lukas@fittl.com>
Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>,
 Tom Lane <tgl@sss.pgh.pa.us>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CA%2BTgmobnwuyTgYXweQrvOz_kFuBuCbKhsLz-mUPasFw_DQ8f6w%40mail.gmail.com>
Precedence: bulk

Hi,

Thanks for taking the time to respond. My reading of your comments is
that we are in overall agreement on the design, with the possible
exception of persisting data cross restarts. I will write more about
that topic below; but I think if that's the only design disagreement
we have, it makes sense to go forward with committing the patch that I
have, and we can continue to discuss whether we want to add something
related to persistence. The only reason not to do that would be if
there were a consensus that the lack of a persistence framework was
such a critical defect that we shouldn't ship this at all without
that, but I don't agree with that idea and I think it would be a
pretty strong position for someone to take.

On Sun, Mar 29, 2026 at 2:59=E2=80=AFPM Lukas Fittl <lukas@fittl.com> wrote=
:
> I think a simple disk file is the way to go, similar to how
> autoprewarm works with its "autoprewarm.blocks" file. Its a bit
> awkward that that just sits in the main data directory, but since
> pg_prewarm already does it today, I think its okay to have another
> contrib module do the same. As noted I'm mainly worried about restarts
> that the user didn't control, causing advice that was set to be lost.
>
> I've attached a patch of how that could look like on top of your v23,
> that copies the modified stash information to a
> "pg_stash_advice.entries" file, and loads it after restarts.

I'll be honest: I don't like this design much at all, but I do see the
practical advantages of it, and we have done similar things elsewhere,
in particular in autoprewarm. Before I get to the specifics of your
patch, let me complain about some things that I don't like at the
design level. We lose a lot by directing data through a bespoke
mechanism rather than handling it as table data. There are no
checksums, so we have less protection against corruption. There is no
write-ahead logging, so data does not make it to standbys, which is
more of a potential issue for pg_stash_advice than it is for
autoprewarm. All the code to read and write the file is specific to
this contrib module, so it can have its own bugs separate from every
similar module's bugs. The data can't easily be examined and
manipulated from SQL as table data can. It's just a messy one-off that
solves a practical problem but is otherwise not very nice. Of course,
sometimes such messy one-offs are the right answer.

In terms of the patch itself, the concurrency situation here seems
noticeably worse than with autoprewarm. In that case, there's only one
authorized writer at a time, tracked via pid_using_dumpfile. But in
this case, it seems like multiple backends could be writing to the
temporary dump file at the same time, which could result in a
corrupted file that doesn't reload properly. Your code also has a race
condition when reloading the data: the first arriving backend tries to
reload the flat file, but any other backends that arrive while that's
in progress see no stashed advice, and if the load fails for some
reason, it's never retried, and the first modification to the
in-memory state will clobber the file. autoprewarm has this issue to
some extent as well, but that's more OK there because recreating the
contents of shared buffers is only an approximate good, but people
probably don't want their stashed advice to disappear out from under
them if it was billed as persistent. That said, I'm not entirely
opposed to a design where there's a small window where the advice
stash is empty after a restart, because avoiding that means that it
has to be safe to do the reload of saved advice from the middle of a
query planning cycle, which is probably true with a flat-file design
but wouldn't be true with a table. Still, I don't know whether the
current behavior is deliberate or accidental.

I also feel a bit uncomfortable with the idea of rewriting the entire
file on every single change. If the hypothesis that this is only for
adjusting the behavior of a small number of critical queries is
correct, then it won't matter, but if people start using this for lots
of queries, it's potentially painful. Neither autoprewarm nor
pg_stat_statements does that. pg_stat_statements reads data only at
postmaster startup and writes data only at postmaster shutdown, so it
simply accepts loss of incremental changes in case of a crash, but
that also means it doesn't read and write the file repeatedly.
autoprewarm writes the file periodically from a background worker so
that the on-disk state doesn't drift too far out of sync with what's
in memory, without promising perfect durability. Both of those
placements have the further advantage that the reading and writing of
the file is not being done "in medias res," which does seem to have
certain advantages from a robustness perspective. For example, without
necessarily endorsing this design, suppose you added a background
worker and there are GUCs to configure the database that it connects
to and the query it executes to restore advice stashes. Or,
alternatively, a background worker that still uses a flat file. Either
way, that opens up design ideas like: when you see that the in-memory
stashes are not yet reloaded, you can decide to wait up to X seconds
for that to happen and then proceed anyway if it hasn't happened by
then. I'm not saying that is the right idea here necessarily, but it's
an option, whereas what you've done doesn't lend itself to that sort
of idea.

One other note is that fscanf() ending in a newline could eat up
whitespace at the start of the following line. Since a stash name can
begin with whitespace, that could be an issue.

> Because the number of entries here is controlled by the user (i.e. its
> not a function of the workload, but a function of how much advice you
> as a user have set), I'm much less worried about memory usage, as long
> as we document it clearly how to measure the amount of memory used.

The module doesn't have a built-in way to do that right now. Are you
thinking we would document that pg_get_dsm_registry_allocations() can
be used?

> In practice for a good amount of our user base these days the question
> will be "Does my cloud provider give me access to create stash
> entries", so its maybe worth thinking about if we could also allow
> pg_maintain to manage entries by default?

Wouldn't it make more sense for the cloud provider to grant execute
permissions on these functions to pg_maintain if they are so inclined?
This is a brand-new facility, so I think we had better be conservative
in terms of default permissions.

--=20
Robert Haas
EDB: http://www.enterprisedb.com