public inbox for [email protected]  

help / color / mirror / Atom feed

From: Tomas Vondra <[email protected]>
To: Michael Paquier <[email protected]>
Cc: wenhui qiu <[email protected]>
Cc: Robert Treat <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: PoC: history of recent vacuum/checkpoint runs (using new hooks)
Date: Tue, 31 Dec 2024 16:06:26 +0100
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
	<CAGjGUAJSg3tJGxjeyU6T98ZYCXcAiGpq0eitAUQVaOa+nULDuA@mail.gmail.com>
	<[email protected]>
	<CABV9wwPcnoS+=27o=pHinCy7fegTTfB=yECyK5XyGecCFfrFHg@mail.gmail.com>
	<CAGjGUAKXjFw+zq+9GZ+y5p360pkO7UuTojkKuB6k3MtdDCEf6w@mail.gmail.com>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>

On 12/31/24 02:06, Michael Paquier wrote:
> On Sat, Dec 28, 2024 at 02:25:16AM +0100, Tomas Vondra wrote:
>> And the more I think about it the more I'm convinced we don't need to
>> keep the data about past runs in memory, a file should be enough (except
>> maybe for a small buffer). That would mean we don't need to worry about
>> dynamic shared memory, etc. I initially rejected this because it seemed
>> like a regression to how pgstat worked initially (sharing data through
>> files), but I don't think that's actually true - this data is different
>> (almost append-only), etc.
> 
> Right, I was looking a bit at 0003 that introduces the extension.  I
> am wondering if we are not repeating the errors of pgss by using a
> different file, and if we should not just use pgstats and its single 
> file instead to store this data through an extension.  You are right
> that as an append-only pattern using the dshash of pgstat does not fit
> well into this picture.   How about the second type of stats kinds:
> the fixed-numbered stats kind?  These allocate a fixed amount of
> shared memory, meaning that you could allocate N entries of history
> and just manage a queue of them, then do a memcpy() of the whole set
> if adding new history at the head of the queue, or just append new
> ones at the tail of the queue in shmem, memcpy() once the queue is
> full.  The extension gets simpler:
> - No need to manage a new file, flush of the stats is controlled by
> pgstats itself.
> - The extension could register a fixed-numbered custom stats kind.


I'm not against leveraging some of the existing pstat infrastructure,
but as I explained earlier I don't like the "fixed amount of shmem"
approach. It's either wasteful (on machines with few vacuum runs) or
difficult to configure to keep enough history.

FWIW I'm not sure what you mean by "pgss errors" - I'm not saying it's
flawless, but OTOH reading/writing a flat file with entries is not
particularly hard. (It'd need to be a bit more complex to evict stuff
for dropped objects, but not by much).

> - Push the stats with the new hooks.
> - Perhaps less useful, but it is possible to control the timing where
> the data is pushed.

Not sure what you mean by "pushed". Pushed where, or how would that matter?

> - Add SQL wrappers on top to fetch the data from pgstat.

OK, although I think this is not particularly complicated part of the
code. It could even be simplified further.


regards

-- 
Tomas Vondra

view thread (10+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected]
  Subject: Re: PoC: history of recent vacuum/checkpoint runs (using new hooks)
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox