pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
pgsql: pgstat: store statistics in shared memory.
5+ messages / 2 participants
[nested] [flat]

* pgsql: pgstat: store statistics in shared memory.
@ 2022-04-07 04:37 Andres Freund <andres@anarazel.de>
  2022-04-07 06:09 ` Re: pgsql: pgstat: store statistics in shared memory. Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 5+ messages in thread

From: Andres Freund @ 2022-04-07 04:37 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

pgstat: store statistics in shared memory.

Previously the statistics collector received statistics updates via UDP and
shared statistics data by writing them out to temporary files regularly. These
files can reach tens of megabytes and are written out up to twice a
second. This has repeatedly prevented us from adding additional useful
statistics.

Now statistics are stored in shared memory. Statistics for variable-numbered
objects are stored in a dshash hashtable (backed by dynamic shared
memory). Fixed-numbered stats are stored in plain shared memory.

The header for pgstat.c contains an overview of the architecture.

The stats collector is not needed anymore, remove it.

By utilizing the transactional statistics drop infrastructure introduced in a
prior commit statistics entries cannot "leak" anymore. Previously leaked
statistics were dropped by pgstat_vacuum_stat(), called from [auto-]vacuum. On
systems with many small relations pgstat_vacuum_stat() could be quite
expensive.

Now that replicas drop statistics entries for dropped objects, it is not
necessary anymore to reset stats when starting from a cleanly shut down
replica.

Subsequent commits will perform some further code cleanup, adapt docs and add
tests.

Bumps PGSTAT_FILE_FORMAT_ID.

Author: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Author: Andres Freund <andres@anarazel.de>
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-By: Andres Freund <andres@anarazel.de>
Reviewed-By: Thomas Munro <thomas.munro@gmail.com>
Reviewed-By: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-By: "David G. Johnston" <david.g.johnston@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@2ndquadrant.com> (in a much earlier version)
Reviewed-By: Arthur Zakirov <a.zakirov@postgrespro.ru> (in a much earlier version)
Reviewed-By: Antonin Houska <ah@cybertec.at> (in a much earlier version)
Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de
Discussion: https://postgr.es/m/20220308205351.2xcn6k4x5yivcxyd@alap3.anarazel.de
Discussion: https://postgr.es/m/20210319235115.y3wz7hpnnrshdyv6@alap3.anarazel.de

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/5891c7a8ed8f2d3d577e7eea34dacff12d7b6bbd

Modified Files
--------------
doc/src/sgml/monitoring.sgml                     |   19 +-
src/backend/access/transam/xlog.c                |   39 +-
src/backend/commands/vacuum.c                    |    7 -
src/backend/commands/vacuumparallel.c            |    2 +
src/backend/postmaster/autovacuum.c              |  197 +-
src/backend/postmaster/checkpointer.c            |   20 +-
src/backend/postmaster/interrupt.c               |    5 +-
src/backend/postmaster/pgstat.c                  | 4637 +++++-----------------
src/backend/postmaster/postmaster.c              |   91 +-
src/backend/replication/logical/logical.c        |    1 -
src/backend/replication/logical/tablesync.c      |    8 +-
src/backend/replication/logical/worker.c         |    6 +
src/backend/replication/slot.c                   |   26 +-
src/backend/storage/buffer/bufmgr.c              |    8 +-
src/backend/storage/ipc/ipci.c                   |    2 +
src/backend/storage/lmgr/lwlock.c                |    8 +-
src/backend/tcop/postgres.c                      |   31 +-
src/backend/utils/activity/Makefile              |    1 +
src/backend/utils/activity/pgstat_archiver.c     |   91 +-
src/backend/utils/activity/pgstat_bgwriter.c     |   82 +-
src/backend/utils/activity/pgstat_checkpointer.c |   93 +-
src/backend/utils/activity/pgstat_database.c     |  345 +-
src/backend/utils/activity/pgstat_function.c     |  167 +-
src/backend/utils/activity/pgstat_relation.c     |  592 +--
src/backend/utils/activity/pgstat_replslot.c     |  183 +-
src/backend/utils/activity/pgstat_shmem.c        |  987 +++++
src/backend/utils/activity/pgstat_slru.c         |  160 +-
src/backend/utils/activity/pgstat_subscription.c |   67 +-
src/backend/utils/activity/pgstat_wal.c          |  175 +-
src/backend/utils/activity/pgstat_xact.c         |   37 +-
src/backend/utils/activity/wait_event.c          |    3 -
src/backend/utils/adt/pgstatfuncs.c              |   10 +-
src/backend/utils/cache/relcache.c               |    7 +-
src/backend/utils/init/globals.c                 |    1 +
src/backend/utils/init/miscinit.c                |    3 -
src/backend/utils/init/postinit.c                |   12 +
src/backend/utils/misc/guc.c                     |   21 +
src/backend/utils/misc/postgresql.conf.sample    |    1 +
src/include/miscadmin.h                          |    2 +-
src/include/pgstat.h                             |  679 +---
src/include/storage/lwlock.h                     |    3 +
src/include/utils/pgstat_internal.h              |  663 +++-
src/include/utils/rel.h                          |    1 +
src/include/utils/timeout.h                      |    1 +
src/include/utils/wait_event.h                   |    1 -
src/test/modules/worker_spi/worker_spi.c         |    2 +-
src/test/regress/expected/stats.out              |    8 +
src/test/regress/sql/stats.sql                   |   10 +
src/tools/pgindent/typedefs.list                 |   63 +-
src/tools/valgrind.supp                          |   18 -
50 files changed, 4253 insertions(+), 5343 deletions(-)



^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: pgsql: pgstat: store statistics in shared memory.
  2022-04-07 04:37 pgsql: pgstat: store statistics in shared memory. Andres Freund <andres@anarazel.de>
@ 2022-04-07 06:09 ` Michael Paquier <michael@paquier.xyz>
  2022-04-07 06:35   ` Re: pgsql: pgstat: store statistics in shared memory. Andres Freund <andres@anarazel.de>
  0 siblings, 1 reply; 5+ messages in thread

From: Michael Paquier @ 2022-04-07 06:09 UTC (permalink / raw)
  To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-committers@lists.postgresql.org

On Thu, Apr 07, 2022 at 04:37:02AM +0000, Andres Freund wrote:
> pgstat: store statistics in shared memory.
> 
> Previously the statistics collector received statistics updates via UDP and
> shared statistics data by writing them out to temporary files regularly. These
> files can reach tens of megabytes and are written out up to twice a
> second. This has repeatedly prevented us from adding additional useful
> statistics.

rorqual, that uses --disable-spinlocks and --disable-atomics, is
unhappy after this commit:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=rorqual&dt=2022-04-07%2005%3A47%3A03

test_decoding is the one complaining here, and I can reproduce the
issue locally once I build the code with those switches.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: pgsql: pgstat: store statistics in shared memory.
  2022-04-07 04:37 pgsql: pgstat: store statistics in shared memory. Andres Freund <andres@anarazel.de>
  2022-04-07 06:09 ` Re: pgsql: pgstat: store statistics in shared memory. Michael Paquier <michael@paquier.xyz>
@ 2022-04-07 06:35   ` Andres Freund <andres@anarazel.de>
  2022-04-07 07:31     ` Re: pgsql: pgstat: store statistics in shared memory. Andres Freund <andres@anarazel.de>
  0 siblings, 1 reply; 5+ messages in thread

From: Andres Freund @ 2022-04-07 06:35 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-committers@lists.postgresql.org

Hi,

On 2022-04-07 15:09:27 +0900, Michael Paquier wrote:
> On Thu, Apr 07, 2022 at 04:37:02AM +0000, Andres Freund wrote:
> > pgstat: store statistics in shared memory.
> > 
> > Previously the statistics collector received statistics updates via UDP and
> > shared statistics data by writing them out to temporary files regularly. These
> > files can reach tens of megabytes and are written out up to twice a
> > second. This has repeatedly prevented us from adding additional useful
> > statistics.
> 
> rorqual, that uses --disable-spinlocks and --disable-atomics, is
> unhappy after this commit:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=rorqual&dt=2022-04-07%2005%3A47%3A03
> 
> test_decoding is the one complaining here, and I can reproduce the
> issue locally once I build the code with those switches.

Yea, saw that too. About to push the fix...

Greetings,

Andres Freund





^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: pgsql: pgstat: store statistics in shared memory.
  2022-04-07 04:37 pgsql: pgstat: store statistics in shared memory. Andres Freund <andres@anarazel.de>
  2022-04-07 06:09 ` Re: pgsql: pgstat: store statistics in shared memory. Michael Paquier <michael@paquier.xyz>
  2022-04-07 06:35   ` Re: pgsql: pgstat: store statistics in shared memory. Andres Freund <andres@anarazel.de>
@ 2022-04-07 07:31     ` Andres Freund <andres@anarazel.de>
  2022-04-07 07:42       ` Re: pgsql: pgstat: store statistics in shared memory. Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 5+ messages in thread

From: Andres Freund @ 2022-04-07 07:31 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-committers@lists.postgresql.org

Hi,

On 2022-04-06 23:35:27 -0700, Andres Freund wrote:
> On 2022-04-07 15:09:27 +0900, Michael Paquier wrote:
> > On Thu, Apr 07, 2022 at 04:37:02AM +0000, Andres Freund wrote:
> > > pgstat: store statistics in shared memory.
> > > 
> > > Previously the statistics collector received statistics updates via UDP and
> > > shared statistics data by writing them out to temporary files regularly. These
> > > files can reach tens of megabytes and are written out up to twice a
> > > second. This has repeatedly prevented us from adding additional useful
> > > statistics.
> > 
> > rorqual, that uses --disable-spinlocks and --disable-atomics, is
> > unhappy after this commit:
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=rorqual&dt=2022-04-07%2005%3A47%3A03
> > 
> > test_decoding is the one complaining here, and I can reproduce the
> > issue locally once I build the code with those switches.
> 
> Yea, saw that too. About to push the fix...

And rorqual seems happy again.

Greetings,

Andres Freund





^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: pgsql: pgstat: store statistics in shared memory.
  2022-04-07 04:37 pgsql: pgstat: store statistics in shared memory. Andres Freund <andres@anarazel.de>
  2022-04-07 06:09 ` Re: pgsql: pgstat: store statistics in shared memory. Michael Paquier <michael@paquier.xyz>
  2022-04-07 06:35   ` Re: pgsql: pgstat: store statistics in shared memory. Andres Freund <andres@anarazel.de>
  2022-04-07 07:31     ` Re: pgsql: pgstat: store statistics in shared memory. Andres Freund <andres@anarazel.de>
@ 2022-04-07 07:42       ` Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 5+ messages in thread

From: Michael Paquier @ 2022-04-07 07:42 UTC (permalink / raw)
  To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-committers@lists.postgresql.org

On Thu, Apr 07, 2022 at 12:31:49AM -0700, Andres Freund wrote:
> And rorqual seems happy again.

Thanks!
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 5+ messages in thread


end of thread, other threads:[~2022-04-07 07:42 UTC | newest]

Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-04-07 04:37 pgsql: pgstat: store statistics in shared memory. Andres Freund <andres@anarazel.de>
2022-04-07 06:09 ` Michael Paquier <michael@paquier.xyz>
2022-04-07 06:35   ` Andres Freund <andres@anarazel.de>
2022-04-07 07:31     ` Andres Freund <andres@anarazel.de>
2022-04-07 07:42       ` Michael Paquier <michael@paquier.xyz>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox