public inbox for [email protected]
help / color / mirror / Atom feedFrom: Greg Stark <[email protected]>
To: Andres Freund <[email protected]>
Cc: Tom Lane <[email protected]>
Cc: Melanie Plageman <[email protected]>
Cc: Kyotaro Horiguchi <[email protected]>
Cc: Justin Pryzby <[email protected]>
Cc: Thomas Munro <[email protected]>
Cc: David G. Johnston <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: shared-memory based stats collector - v70
Date: Thu, 21 Jul 2022 11:07:55 -0400
Message-ID: <CAM-w4HMhHAxjpXNVMZ5WhULHpd3+LuP27zaSKSFnzZeRwHZTjg@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<CAAKRu_YsFzW3jsQUD7MVUY8c+87OiKW_Y93TsPAfLr3zi_jX9w@mail.gmail.com>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<CAM-w4HN=4ss5dS2DH6Cof=zocJXeGOjRNKxEGgUZdJPMndyNzA@mail.gmail.com>
<CAAKRu_Z18jYbarKA4GZMOO00KtUqKtBqgQ8SgB=wSvSqKq3iiA@mail.gmail.com>
<[email protected]>
<CAM-w4HOxaO6iJ94bGt1A65PRgX_b4gGSmb4mdzakZ1QhvGW=jA@mail.gmail.com>
<[email protected]>
On Wed, 20 Jul 2022 at 15:09, Andres Freund <[email protected]> wrote:
>
> Each backend only had stats for things it touched. But the stats collector read all files at startup into hash tables and absorbed all generated stats into those as well.
Fascinating. I'm surprised this didn't raise issues previously for
people with millions of tables. I wonder if it wasn't causing issues
and we just didn't hear about them because there were other bigger
issues :)
> >All that said -- having all objects loaded in shared memory makes my
> >work way easier.
>
> What are your trying to do?
I'm trying to implement an exporter for prometheus/openmetrics/etc
that dumps directly from shared memory without going through the SQL
backend layer. I believe this will be much more reliable, lower
overhead, safer, and consistent than writing SQL queries.
Ideally I would want to dump out the stats without connecting to each
database. I suspect that would run into problems where the schema
really adds a lot of information (such as which table each index is on
or which table a toast relation is for. There are also some things
people think of as stats that are maintained in the catalog such as
reltuples and relpages. So I'm imagining this won't strictly stay true
in the end.
It seems like just having an interface to iterate over the shared hash
table and return entries one by one without filtering by database
would be fairly straightforward and I would be able to do most of what
I want just with that. There's actually enough meta information in the
stats entries to be able to handle them as they come instead of trying
to process look up specific stats one by one.
--
greg
view thread (82+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: shared-memory based stats collector - v70
In-Reply-To: <CAM-w4HMhHAxjpXNVMZ5WhULHpd3+LuP27zaSKSFnzZeRwHZTjg@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox