Feedback-ID: id4a34324:Fastmail
Date: Thu, 18 Aug 2022 12:51:24 -0700
From: Andres Freund <andres@anarazel.de>
To: Greg Stark <stark@mit.edu>
Cc: "Drouvot, Bertrand" <bdrouvot@amazon.com>, Tom Lane <tgl@sss.pgh.pa.us>,
	Melanie Plageman <melanieplageman@gmail.com>,
	Kyotaro Horiguchi <horikyota.ntt@gmail.com>,
	Justin Pryzby <pryzby@telsasoft.com>,
	Thomas Munro <thomas.munro@gmail.com>,
	"David G. Johnston" <david.g.johnston@gmail.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: shared-memory based stats collector - v70
Message-ID: <20220818195124.c7ipzf6c5v7vxymc@awork3.anarazel.de>
References: <caa515eb-e9e7-9fe7-ceb1-a96b6f3787ce@amazon.com>
 <CAM-w4HNqCR7X9q5_wbJi4wLfTT1gLvy5VRS+ZGo6z8-DbXkHpw@mail.gmail.com>
 <5db2b670-f3df-57ea-bdc6-962a582ff2e4@amazon.com>
 <CAM-w4HP_YL8uPrNiv2DPKmq1KD3QLEX7CBLc5DCJ9j7mf7eerw@mail.gmail.com>
 <0befffb3-8323-f2cd-915f-0ab64396f500@amazon.com>
 <CAM-w4HPGq_rpZAW4p71+d90oykyi9y7PgiQ1c9_ROs+WROCjuQ@mail.gmail.com>
 <1c56f4c6-61ce-d131-553c-297326e20bf1@amazon.com>
 <CAM-w4HNLtcKxrG0WepCKjt5tizbvZRcrqABX=jNNZoi_18E5rw@mail.gmail.com>
 <7f622949-2d78-c030-e7e0-cdee5f6cf79b@amazon.com>
 <CAM-w4HNm06PcCM40-kY+jLgD_Tqjr3ymBy8qErfOvWqQyJE+6Q@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CAM-w4HNm06PcCM40-kY+jLgD_Tqjr3ymBy8qErfOvWqQyJE+6Q@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/20220818195124.c7ipzf6c5v7vxymc%40awork3.anarazel.de>
Precedence: bulk

Hi,

On 2022-08-18 15:26:31 -0400, Greg Stark wrote:
> And indexes of course. It's a bit frustrating since without the
> catalog you won't know what table the index actually is for... But
> they're pretty important stats.

FWIW, I think we should split relation stats into table and index
stats. Historically it'd have added a lot of complexity to separate the two,
but I don't think that's the case anymore. And we waste space for index stats
by having lots of table specific fields.


> On that note though... What do you think about having the capability
> to add other stats kinds to the stats infrastructure?

Getting closer to that was one of my goals working on the shared memory stats
stuff.


> It would make a lot of sense for pg_stat_statements to add its entries here
> instead of having to reimplement a lot of the same magic.

Yes, we should move pg_stat_statements over.

It's pretty easy to get massive contention on stats entries with
pg_stat_statements, because it doesn't have support for "batching" updates to
shared stats. And reimplementing the same logic in pg_stat_statements.c
doesn't make sense.

And the set of normalized queries could probably stored in DSA as well - the
file based thing we have right now is problematic.


> To do that I guess more of the code needs to be moved to be table
> driven from the kind structs either with callbacks or with other meta
> data.

Pretty much all of it already is. The only substantial missing bit is
reading/writing of stats files, but that should be pretty easy. And of course
making the callback array extensible.


> So the kind record could contain tupledesc and the code to construct the
> returned tuple so that these functions could return any custom entry as well
> as the standard entries.

I don't see how this would work well - we don't have functions returning
variable kinds of tuples. And what would convert a struct to a tuple?

Nor do I think it's needed - if you have an extension providing a new stats
kind it can also provide accessors.

Greetings,

Andres Freund