public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v50 4/7] Shared-memory based stats collector
5+ messages / 4 participants
[nested] [flat]
* [PATCH v50 4/7] Shared-memory based stats collector
@ 2021-03-09 06:07 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-09 06:07 UTC (permalink / raw)
Previously activity statistics is collected via sockets and shared
among backends through files periodically. Such files reaches tens of
megabytes and are created at most every 1 second and such large data
is serialized by stats collector then de-serialized on every backend
periodically. To evade that large cost, this patch places activity
statistics data on shared memory. Each backend accumulates statistics
numbers locally then tries to move them onto the shared statistics at
every transaction end but with intervals not shorter than 10 seconds.
Until 60 second has elapsed since the last flushing to shared stats,
lock failure postpones stats flushing to try to alleviate lock
contention that slows down transactions. Finally stats flush waits
for locks so that shared statistics doesn't get stale.
---
src/backend/access/heap/heapam_handler.c | 4 +-
src/backend/access/heap/vacuumlazy.c | 2 +-
src/backend/access/transam/xlog.c | 14 +-
src/backend/catalog/index.c | 24 +-
src/backend/commands/analyze.c | 8 +-
src/backend/commands/dbcommands.c | 2 +-
src/backend/commands/matview.c | 8 +-
src/backend/commands/vacuum.c | 4 +-
src/backend/postmaster/autovacuum.c | 76 +-
src/backend/postmaster/bgwriter.c | 4 +-
src/backend/postmaster/checkpointer.c | 26 +-
src/backend/postmaster/pgarch.c | 12 +-
src/backend/postmaster/pgstat.c | 6374 +++++++----------
src/backend/postmaster/postmaster.c | 82 +-
src/backend/postmaster/walwriter.c | 2 +-
src/backend/replication/basebackup.c | 4 +-
src/backend/replication/slot.c | 12 +-
src/backend/storage/buffer/bufmgr.c | 8 +-
src/backend/storage/ipc/ipci.c | 2 +
src/backend/storage/lmgr/lwlock.c | 4 +-
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/storage/smgr/smgr.c | 4 +-
src/backend/tcop/postgres.c | 41 +-
src/backend/utils/adt/pgstatfuncs.c | 109 +-
src/backend/utils/cache/relcache.c | 5 +
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/miscinit.c | 3 -
src/backend/utils/init/postinit.c | 11 +
src/backend/utils/misc/guc.c | 16 +-
src/backend/utils/misc/postgresql.conf.sample | 2 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 4 +-
src/include/miscadmin.h | 3 +-
src/include/pgstat.h | 760 +-
src/include/storage/lwlock.h | 1 +
src/include/utils/guc_tables.h | 2 +-
src/include/utils/timeout.h | 1 +
src/tools/pgindent/typedefs.list | 52 +-
37 files changed, 2919 insertions(+), 4769 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index bd5faf0c1f..c88339282f 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1086,8 +1086,8 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* our own. In this case we should count and sample the row,
* to accommodate users who load a table and analyze it in one
* transaction. (pgstat_report_analyze has to adjust the
- * numbers we send to the stats collector to make this come
- * out right.)
+ * numbers we report to the activity stats facility to make
+ * this come out right.)
*/
if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetXmin(targtuple->t_data)))
{
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d8f847b0e6..4e90b4bdda 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -598,7 +598,7 @@ heap_vacuum_rel(Relation onerel, VacuumParams *params,
new_min_multi,
false);
- /* report results to the stats collector, too */
+ /* report results to the activity stats facility, too */
pgstat_report_vacuum(RelationGetRelid(onerel),
onerel->rd_rel->relisshared,
Max(new_live_tuples, 0),
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 18af3d4120..2a7d995320 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -2204,7 +2204,7 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, bool opportunistic)
WriteRqst.Flush = 0;
XLogWrite(WriteRqst, false);
LWLockRelease(WALWriteLock);
- WalStats.m_wal_buffers_full++;
+ WalStats.wal_buffers_full++;
TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE();
}
/* Re-acquire WALBufMappingLock and retry */
@@ -2562,10 +2562,10 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible)
INSTR_TIME_SET_CURRENT(duration);
INSTR_TIME_SUBTRACT(duration, start);
- WalStats.m_wal_write_time += INSTR_TIME_GET_MICROSEC(duration);
+ WalStats.wal_write_time += INSTR_TIME_GET_MICROSEC(duration);
}
- WalStats.m_wal_write++;
+ WalStats.wal_write++;
if (written <= 0)
{
@@ -8665,8 +8665,8 @@ LogCheckpointEnd(bool restartpoint)
CheckpointStats.ckpt_sync_end_t);
/* Accumulate checkpoint timing summary data, in milliseconds. */
- BgWriterStats.m_checkpoint_write_time += write_msecs;
- BgWriterStats.m_checkpoint_sync_time += sync_msecs;
+ CheckPointerStats.checkpoint_write_time += write_msecs;
+ CheckPointerStats.checkpoint_sync_time += sync_msecs;
/*
* All of the published timing statistics are accounted for. Only
@@ -10616,10 +10616,10 @@ issue_xlog_fsync(int fd, XLogSegNo segno)
INSTR_TIME_SET_CURRENT(duration);
INSTR_TIME_SUBTRACT(duration, start);
- WalStats.m_wal_sync_time += INSTR_TIME_GET_MICROSEC(duration);
+ WalStats.wal_sync_time += INSTR_TIME_GET_MICROSEC(duration);
}
- WalStats.m_wal_sync++;
+ WalStats.wal_sync++;
}
/*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..85f8d32944 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1859,28 +1859,10 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
/*
* Copy over statistics from old to new index
+ * The data will be flushed by the next pgstat_report_stat()
+ * call.
*/
- {
- PgStat_StatTabEntry *tabentry;
-
- tabentry = pgstat_fetch_stat_tabentry(oldIndexId);
- if (tabentry)
- {
- if (newClassRel->pgstat_info)
- {
- newClassRel->pgstat_info->t_counts.t_numscans = tabentry->numscans;
- newClassRel->pgstat_info->t_counts.t_tuples_returned = tabentry->tuples_returned;
- newClassRel->pgstat_info->t_counts.t_tuples_fetched = tabentry->tuples_fetched;
- newClassRel->pgstat_info->t_counts.t_blocks_fetched = tabentry->blocks_fetched;
- newClassRel->pgstat_info->t_counts.t_blocks_hit = tabentry->blocks_hit;
-
- /*
- * The data will be sent by the next pgstat_report_stat()
- * call.
- */
- }
- }
- }
+ pgstat_copy_index_counters(oldIndexId, newClassRel->pgstat_info);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 7295cf0215..308b4ab034 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -644,10 +644,10 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
}
/*
- * Report ANALYZE to the stats collector, too. However, if doing
- * inherited stats we shouldn't report, because the stats collector only
- * tracks per-table stats. Reset the changes_since_analyze counter only
- * if we analyzed all columns; otherwise, there is still work for
+ * Report ANALYZE to the activity stats facility, too. However, if doing
+ * inherited stats we shouldn't report, because the activity stats facility
+ * only tracks per-table stats. Reset the changes_since_analyze counter
+ * only if we analyzed all columns; otherwise, there is still work for
* auto-analyze to do.
*/
if (!inh)
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 2b159b60eb..acea4de382 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -971,7 +971,7 @@ dropdb(const char *dbname, bool missing_ok, bool force)
DropDatabaseBuffers(db_id);
/*
- * Tell the stats collector to forget it immediately, too.
+ * Tell the active stats facility to forget it immediately, too.
*/
pgstat_drop_database(db_id);
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index c5c25ce11d..1464b97c7f 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -336,10 +336,10 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString,
refresh_by_heap_swap(matviewOid, OIDNewHeap, relpersistence);
/*
- * Inform stats collector about our activity: basically, we truncated
- * the matview and inserted some new data. (The concurrent code path
- * above doesn't need to worry about this because the inserts and
- * deletes it issues get counted by lower-level code.)
+ * Inform activity stats facility about our activity: basically, we
+ * truncated the matview and inserted some new data. (The concurrent
+ * code path above doesn't need to worry about this because the inserts
+ * and deletes it issues get counted by lower-level code.)
*/
pgstat_count_truncate(matviewRel);
if (!stmt->skipData)
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index c064352e23..7397c3704d 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -329,8 +329,8 @@ vacuum(List *relations, VacuumParams *params,
errmsg("PROCESS_TOAST required with VACUUM FULL")));
/*
- * Send info about dead objects to the statistics collector, unless we are
- * in autovacuum --- autovacuum.c does this for itself.
+ * Send info about dead objects to the activity statistics facility, unless
+ * we are in autovacuum --- autovacuum.c does this for itself.
*/
if ((params->options & VACOPT_VACUUM) && !IsAutoVacuumWorkerProcess())
pgstat_vacuum_stat();
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 23ef23c13e..46f969bef3 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -342,9 +342,6 @@ static void autovacuum_do_vac_analyze(autovac_table *tab,
BufferAccessStrategy bstrategy);
static AutoVacOpts *extract_autovac_opts(HeapTuple tup,
TupleDesc pg_class_desc);
-static PgStat_StatTabEntry *get_pgstat_tabentry_relid(Oid relid, bool isshared,
- PgStat_StatDBEntry *shared,
- PgStat_StatDBEntry *dbentry);
static void perform_work_item(AutoVacuumWorkItem *workitem);
static void autovac_report_activity(autovac_table *tab);
static void autovac_report_workitem(AutoVacuumWorkItem *workitem,
@@ -1684,12 +1681,12 @@ AutoVacWorkerMain(int argc, char *argv[])
char dbname[NAMEDATALEN];
/*
- * Report autovac startup to the stats collector. We deliberately do
- * this before InitPostgres, so that the last_autovac_time will get
- * updated even if the connection attempt fails. This is to prevent
- * autovac from getting "stuck" repeatedly selecting an unopenable
- * database, rather than making any progress on stuff it can connect
- * to.
+ * Report autovac startup to the activity stats facility. We
+ * deliberately do this before InitPostgres, so that the
+ * last_autovac_time will get updated even if the connection attempt
+ * fails. This is to prevent autovac from getting "stuck" repeatedly
+ * selecting an unopenable database, rather than making any progress on
+ * stuff it can connect to.
*/
pgstat_report_autovac(dbid);
@@ -1961,8 +1958,6 @@ do_autovacuum(void)
HASHCTL ctl;
HTAB *table_toast_map;
ListCell *volatile cell;
- PgStat_StatDBEntry *shared;
- PgStat_StatDBEntry *dbentry;
BufferAccessStrategy bstrategy;
ScanKeyData key;
TupleDesc pg_class_desc;
@@ -1981,17 +1976,11 @@ do_autovacuum(void)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(AutovacMemCxt);
- /*
- * may be NULL if we couldn't find an entry (only happens if we are
- * forcing a vacuum for anti-wrap purposes).
- */
- dbentry = pgstat_fetch_stat_dbentry(MyDatabaseId);
-
/* Start a transaction so our commands have one to play into. */
StartTransactionCommand();
/*
- * Clean up any dead statistics collector entries for this DB. We always
+ * Clean up any dead activity statistics entries for this DB. We always
* want to do this exactly once per DB-processing cycle, even if we find
* nothing worth vacuuming in the database.
*/
@@ -2034,9 +2023,6 @@ do_autovacuum(void)
/* StartTransactionCommand changed elsewhere */
MemoryContextSwitchTo(AutovacMemCxt);
- /* The database hash where pgstat keeps shared relations */
- shared = pgstat_fetch_stat_dbentry(InvalidOid);
-
classRel = table_open(RelationRelationId, AccessShareLock);
/* create a copy so we can use it after closing pg_class */
@@ -2114,8 +2100,8 @@ do_autovacuum(void)
/* Fetch reloptions and the pgstat entry for this table */
relopts = extract_autovac_opts(tuple, pg_class_desc);
- tabentry = get_pgstat_tabentry_relid(relid, classForm->relisshared,
- shared, dbentry);
+ tabentry = pgstat_fetch_stat_tabentry_extended(classForm->relisshared,
+ relid);
/* Check if it needs vacuum or analyze */
relation_needs_vacanalyze(relid, relopts, classForm, tabentry,
@@ -2198,8 +2184,8 @@ do_autovacuum(void)
}
/* Fetch the pgstat entry for this table */
- tabentry = get_pgstat_tabentry_relid(relid, classForm->relisshared,
- shared, dbentry);
+ tabentry = pgstat_fetch_stat_tabentry_extended(classForm->relisshared,
+ relid);
relation_needs_vacanalyze(relid, relopts, classForm, tabentry,
effective_multixact_freeze_max_age,
@@ -2758,29 +2744,6 @@ extract_autovac_opts(HeapTuple tup, TupleDesc pg_class_desc)
return av;
}
-/*
- * get_pgstat_tabentry_relid
- *
- * Fetch the pgstat entry of a table, either local to a database or shared.
- */
-static PgStat_StatTabEntry *
-get_pgstat_tabentry_relid(Oid relid, bool isshared, PgStat_StatDBEntry *shared,
- PgStat_StatDBEntry *dbentry)
-{
- PgStat_StatTabEntry *tabentry = NULL;
-
- if (isshared)
- {
- if (PointerIsValid(shared))
- tabentry = hash_search(shared->tables, &relid,
- HASH_FIND, NULL);
- }
- else if (PointerIsValid(dbentry))
- tabentry = hash_search(dbentry->tables, &relid,
- HASH_FIND, NULL);
-
- return tabentry;
-}
/*
* table_recheck_autovac
@@ -2985,17 +2948,10 @@ recheck_relation_needs_vacanalyze(Oid relid,
bool *wraparound)
{
PgStat_StatTabEntry *tabentry;
- PgStat_StatDBEntry *shared = NULL;
- PgStat_StatDBEntry *dbentry = NULL;
-
- if (classForm->relisshared)
- shared = pgstat_fetch_stat_dbentry(InvalidOid);
- else
- dbentry = pgstat_fetch_stat_dbentry(MyDatabaseId);
/* fetch the pgstat table entry */
- tabentry = get_pgstat_tabentry_relid(relid, classForm->relisshared,
- shared, dbentry);
+ tabentry = pgstat_fetch_stat_tabentry_extended(classForm->relisshared,
+ relid);
relation_needs_vacanalyze(relid, avopts, classForm, tabentry,
effective_multixact_freeze_max_age,
@@ -3025,7 +2981,7 @@ recheck_relation_needs_vacanalyze(Oid relid,
*
* For analyze, the analysis done is that the number of tuples inserted,
* deleted and updated since the last analyze exceeds a threshold calculated
- * in the same fashion as above. Note that the collector actually stores
+ * in the same fashion as above. Note that the activity statistics stores
* the number of tuples (both live and dead) that there were as of the last
* analyze. This is asymmetric to the VACUUM case.
*
@@ -3035,8 +2991,8 @@ recheck_relation_needs_vacanalyze(Oid relid,
*
* A table whose autovacuum_enabled option is false is
* automatically skipped (unless we have to vacuum it due to freeze_max_age).
- * Thus autovacuum can be disabled for specific tables. Also, when the stats
- * collector does not have data about a table, it will be skipped.
+ * Thus autovacuum can be disabled for specific tables. Also, when the activity
+ * statistics does not have data about a table, it will be skipped.
*
* A table whose vac_base_thresh value is < 0 takes the base value from the
* autovacuum_vacuum_threshold GUC variable. Similarly, a vac_scale_factor
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 715d5195bb..679992dc89 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -244,9 +244,9 @@ BackgroundWriterMain(void)
can_hibernate = BgBufferSync(&wb_context);
/*
- * Send off activity statistics to the stats collector
+ * Send off activity statistics to the activity stats facility
*/
- pgstat_send_bgwriter();
+ pgstat_report_bgwriter();
if (FirstCallSinceLastCheckpoint())
{
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 57c4d5a5d9..4f8773abff 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -360,7 +360,7 @@ CheckpointerMain(void)
if (((volatile CheckpointerShmemStruct *) CheckpointerShmem)->ckpt_flags)
{
do_checkpoint = true;
- BgWriterStats.m_requested_checkpoints++;
+ CheckPointerStats.requested_checkpoints++;
}
/*
@@ -374,7 +374,7 @@ CheckpointerMain(void)
if (elapsed_secs >= CheckPointTimeout)
{
if (!do_checkpoint)
- BgWriterStats.m_timed_checkpoints++;
+ CheckPointerStats.timed_checkpoints++;
do_checkpoint = true;
flags |= CHECKPOINT_CAUSE_TIME;
}
@@ -495,16 +495,10 @@ CheckpointerMain(void)
/* Check for archive_timeout and switch xlog files if necessary. */
CheckArchiveTimeout();
- /*
- * Send off activity statistics to the stats collector. (The reason
- * why we re-use bgwriter-related code for this is that the bgwriter
- * and checkpointer used to be just one process. It's probably not
- * worth the trouble to split the stats support into two independent
- * stats message types.)
- */
- pgstat_send_bgwriter();
+ /* Send off activity statistics to the activity stats facility. */
+ pgstat_report_checkpointer();
- /* Send WAL statistics to the stats collector. */
+ /* Send off WAL statistics to the activity stats facility. */
pgstat_report_wal();
/*
@@ -711,9 +705,9 @@ CheckpointWriteDelay(int flags, double progress)
CheckArchiveTimeout();
/*
- * Report interim activity statistics to the stats collector.
+ * Report interim activity statistics.
*/
- pgstat_send_bgwriter();
+ pgstat_report_checkpointer();
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
@@ -1257,8 +1251,10 @@ AbsorbSyncRequests(void)
LWLockAcquire(CheckpointerCommLock, LW_EXCLUSIVE);
/* Transfer stats counts into pending pgstats message */
- BgWriterStats.m_buf_written_backend += CheckpointerShmem->num_backend_writes;
- BgWriterStats.m_buf_fsync_backend += CheckpointerShmem->num_backend_fsync;
+ CheckPointerStats.buf_written_backend
+ += CheckpointerShmem->num_backend_writes;
+ CheckPointerStats.buf_fsync_backend
+ += CheckpointerShmem->num_backend_fsync;
CheckpointerShmem->num_backend_writes = 0;
CheckpointerShmem->num_backend_fsync = 0;
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index e237cedaff..1f5de65ca2 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -330,20 +330,20 @@ pgarch_ArchiverCopyLoop(void)
pgarch_archiveDone(xlog);
/*
- * Tell the collector about the WAL file that we successfully
- * archived
+ * Tell the activity statistics facility about the WAL file
+ * that we successfully archived
*/
- pgstat_send_archiver(xlog, false);
+ pgstat_report_archiver(xlog, false);
break; /* out of inner retry loop */
}
else
{
/*
- * Tell the collector about the WAL file that we failed to
- * archive
+ * Tell the activity statistics facility about the WAL file
+ * that we failed to archive
*/
- pgstat_send_archiver(xlog, true);
+ pgstat_report_archiver(xlog, true);
if (++failures >= NUM_ARCHIVE_RETRIES)
{
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 05d5ce0064..e755698ea6 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -1,15 +1,22 @@
/* ----------
* pgstat.c
*
- * All the statistics collector stuff hacked up in one big, ugly file.
+ * Activity Statistics facility.
*
- * TODO: - Separate collector, postmaster and backend stuff
- * into different files.
+ * Collects activity statistics, e.g. per-table access statistics, of
+ * all backends in shared memory. The activity numbers are first stored
+ * locally in each process, then flushed to shared memory at commit
+ * time or by idle-timeout.
*
- * - Add some automatic call for pgstat vacuuming.
+ * To avoid congestion on the shared memory, shared stats is updated no more
+ * often than once per PGSTAT_MIN_INTERVAL (10000ms). If some local numbers
+ * remain unflushed for lock failure, retry with intervals that is initially
+ * PGSTAT_RETRY_MIN_INTERVAL (1000ms) then doubled at every retry. Finally we
+ * force update after PGSTAT_MAX_INTERVAL (60000ms) since the first trial.
*
- * - Add a pgstat config column to pg_database, so this
- * entire thing can be enabled/disabled on a per db basis.
+ * The first process that uses activity statistics facility creates the area
+ * then load the stored stats file if any, and the last process at shutdown
+ * writes the shared stats to the file then destroy the area before exit.
*
* Copyright (c) 2001-2021, PostgreSQL Global Development Group
*
@@ -19,18 +26,6 @@
#include "postgres.h"
#include <unistd.h>
-#include <fcntl.h>
-#include <sys/param.h>
-#include <sys/time.h>
-#include <sys/socket.h>
-#include <netdb.h>
-#include <netinet/in.h>
-#include <arpa/inet.h>
-#include <signal.h>
-#include <time.h>
-#ifdef HAVE_SYS_SELECT_H
-#include <sys/select.h>
-#endif
#include "access/heapam.h"
#include "access/htup_details.h"
@@ -40,13 +35,9 @@
#include "access/xact.h"
#include "catalog/pg_database.h"
#include "catalog/pg_proc.h"
-#include "common/ip.h"
-#include "executor/instrument.h"
+#include "common/hashfn.h"
#include "libpq/libpq.h"
-#include "libpq/pqsignal.h"
-#include "mb/pg_wchar.h"
#include "miscadmin.h"
-#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
#include "postmaster/fork_process.h"
@@ -54,20 +45,16 @@
#include "postmaster/postmaster.h"
#include "replication/slot.h"
#include "replication/walsender.h"
-#include "storage/backendid.h"
-#include "storage/dsm.h"
-#include "storage/fd.h"
+#include "storage/condition_variable.h"
#include "storage/ipc.h"
-#include "storage/latch.h"
#include "storage/lmgr.h"
-#include "storage/pg_shmem.h"
+#include "storage/proc.h"
#include "storage/procsignal.h"
#include "storage/sinvaladt.h"
#include "utils/ascii.h"
#include "utils/guc.h"
#include "utils/memutils.h"
-#include "utils/ps_status.h"
-#include "utils/rel.h"
+#include "utils/probes.h"
#include "utils/snapmgr.h"
#include "utils/timestamp.h"
@@ -75,35 +62,20 @@
* Timer definitions.
* ----------
*/
-#define PGSTAT_STAT_INTERVAL 500 /* Minimum time between stats file
- * updates; in milliseconds. */
+#define PGSTAT_MIN_INTERVAL 10000 /* Minimum interval of stats data
+ * updates; in milliseconds. */
-#define PGSTAT_RETRY_DELAY 10 /* How long to wait between checks for a
- * new file; in milliseconds. */
-
-#define PGSTAT_MAX_WAIT_TIME 10000 /* Maximum time to wait for a stats
- * file update; in milliseconds. */
-
-#define PGSTAT_INQ_INTERVAL 640 /* How often to ping the collector for a
- * new file; in milliseconds. */
-
-#define PGSTAT_RESTART_INTERVAL 60 /* How often to attempt to restart a
- * failed statistics collector; in
- * seconds. */
-
-#define PGSTAT_POLL_LOOP_COUNT (PGSTAT_MAX_WAIT_TIME / PGSTAT_RETRY_DELAY)
-#define PGSTAT_INQ_LOOP_COUNT (PGSTAT_INQ_INTERVAL / PGSTAT_RETRY_DELAY)
-
-/* Minimum receive buffer size for the collector's socket. */
-#define PGSTAT_MIN_RCVBUF (100 * 1024)
+#define PGSTAT_RETRY_MIN_INTERVAL 1000 /* Initial retry interval after
+ * PGSTAT_MIN_INTERVAL */
+#define PGSTAT_MAX_INTERVAL 60000 /* Longest interval of stats data
+ * updates */
/* ----------
- * The initial size hints for the hash tables used in the collector.
+ * The initial size hints for the hash tables used in the activity statistics.
* ----------
*/
-#define PGSTAT_DB_HASH_SIZE 16
-#define PGSTAT_TAB_HASH_SIZE 512
+#define PGSTAT_TABLE_HASH_SIZE 512
#define PGSTAT_FUNCTION_HASH_SIZE 512
@@ -118,7 +90,6 @@
*/
#define NumBackendStatSlots (MaxBackends + NUM_AUXPROCTYPES)
-
/* ----------
* GUC parameters
* ----------
@@ -132,17 +103,11 @@ int pgstat_track_activity_query_size = 1024;
* Built from GUC parameter
* ----------
*/
-char *pgstat_stat_directory = NULL;
-char *pgstat_stat_filename = NULL;
-char *pgstat_stat_tmpname = NULL;
+char *pgstat_stat_directory = NULL;
-/*
- * BgWriter and WAL global statistics counters.
- * Stored directly in a stats message structure so they can be sent
- * without needing to copy things around. We assume these init to zeroes.
- */
-PgStat_MsgBgWriter BgWriterStats;
-PgStat_MsgWal WalStats;
+/* No longer used, but will be removed with GUC */
+char *pgstat_stat_filename = NULL;
+char *pgstat_stat_tmpname = NULL;
/*
* WAL usage counters saved from pgWALUsage at the previous call to
@@ -170,73 +135,247 @@ static const char *const slru_names[] = {
#define SLRU_NUM_ELEMENTS lengthof(slru_names)
+/* struct for shared SLRU stats */
+typedef struct PgStatSharedSLRUStats
+{
+ PgStat_SLRUStats entry[SLRU_NUM_ELEMENTS];
+ LWLock lock;
+ pg_atomic_uint32 changecount;
+} PgStatSharedSLRUStats;
+
+StaticAssertDecl(sizeof(TimestampTz) == sizeof(pg_atomic_uint64),
+ "size of pg_atomic_uint64 doesn't match TimestampTz");
+
+typedef struct StatsShmemStruct
+{
+ dsa_handle stats_dsa_handle; /* handle for stats data area */
+ dshash_table_handle hash_handle; /* shared dbstat hash */
+ int refcount; /* # of processes that is attaching the shared
+ * stats memory */
+ /* Global stats structs */
+ PgStat_Archiver archiver_stats;
+ pg_atomic_uint32 archiver_changecount;
+ PgStat_BgWriter bgwriter_stats;
+ pg_atomic_uint32 bgwriter_changecount;
+ PgStat_CheckPointer checkpointer_stats;
+ pg_atomic_uint32 checkpointer_changecount;
+ PgStat_Wal wal_stats;
+ LWLock wal_stats_lock;
+ PgStatSharedSLRUStats slru_stats;
+ pg_atomic_uint32 slru_changecount;
+ pg_atomic_uint64 stats_timestamp;
+
+ /* Reset offsets, protected by StatsLock */
+ PgStat_Archiver archiver_reset_offset;
+ PgStat_BgWriter bgwriter_reset_offset;
+ PgStat_CheckPointer checkpointer_reset_offset;
+
+ /* file read/write protection */
+ bool attach_holdoff;
+ ConditionVariable holdoff_cv;
+
+ pg_atomic_uint64 gc_count; /* # of entries deleted. not protected by
+ * StatsLock */
+} StatsShmemStruct;
+
+/* BgWriter global statistics counters */
+PgStat_BgWriter BgWriterStats = {0};
+
+/* CheckPointer global statistics counters */
+PgStat_CheckPointer CheckPointerStats = {0};
+
+/* WAL global statistics counters */
+PgStat_Wal WalStats = {0};
+
/*
- * SLRU statistics counts waiting to be sent to the collector. These are
- * stored directly in stats message format so they can be sent without needing
- * to copy things around. We assume this variable inits to zeroes. Entries
- * are one-to-one with slru_names[].
+ * XXXX: always try to flush WAL stats. We don't want to manipulate another
+ * counter during XLogInsert so we don't have an effecient short cut to know
+ * whether any counter gets incremented.
*/
-static PgStat_MsgSLRU SLRUStats[SLRU_NUM_ELEMENTS];
+static inline bool
+walstats_pending(void)
+{
+ static const PgStat_Wal all_zeroes;
+
+ return memcmp(&WalStats, &all_zeroes,
+ offsetof(PgStat_Wal, stat_reset_timestamp)) != 0;
+}
+
+/*
+ * SLRU statistics counts waiting to be written to the shared activity
+ * statistics. We assume this variable inits to zeroes. Entries are
+ * one-to-one with slru_names[].
+ * Changes of SLRU counters are reported within critical sections so we use
+ * static memory in order to avoid memory allocation.
+ */
+static PgStat_SLRUStats local_SLRUStats[SLRU_NUM_ELEMENTS];
+static bool have_slrustats = false;
/* ----------
* Local data
* ----------
*/
-NON_EXEC_STATIC pgsocket pgStatSock = PGINVALID_SOCKET;
-
-static struct sockaddr_storage pgStatAddr;
-
-static time_t last_pgstat_start_time;
-
-static bool pgStatRunningInCollector = false;
+/* backend-lifetime storages */
+static StatsShmemStruct *StatsShmem = NULL;
+static dsa_area *area = NULL;
/*
- * Structures in which backends store per-table info that's waiting to be
- * sent to the collector.
- *
- * NOTE: once allocated, TabStatusArray structures are never moved or deleted
- * for the life of the backend. Also, we zero out the t_id fields of the
- * contained PgStat_TableStatus structs whenever they are not actively in use.
- * This allows relcache pgstat_info pointers to be treated as long-lived data,
- * avoiding repeated searches in pgstat_initstats() when a relation is
- * repeatedly opened during a transaction.
+ * Types to define shared statistics structure.
+ *
+ * Per-object statistics are stored in a "shared stats", corresponding struct
+ * that has a header part common among all object types in DSA-allocated
+ * memory. All shared stats are pointed from a dshash via a dsa_pointer. This
+ * structure make the shared stats immovable against dshash resizing, allows a
+ * backend point to shared stats entries via a native pointer and allows
+ * locking at stats-entry level. The per-entry locking reduces lock contention
+ * compared to partition lock of dshash. A backend accumulates stats numbers in
+ * a stats entry in the local memory space then flushes the numbers to shared
+ * stats entries at basically transaction end.
+ *
+ * Each stat entry type has a fixed member PgStat_HashEntryHeader as the first
+ * element.
+ *
+ * Shared stats are stored as:
+ *
+ * dshash pgStatSharedHash
+ * -> PgStatHashEntry (dshash entry)
+ * (dsa_pointer)-> PgStat_Stat*Entry (dsa memory block)
+ *
+ * Shared stats entries are directly pointed from pgstat_localhash hash:
+ *
+ * pgstat_localhash pgStatEntHash
+ * -> PgStatLocalHashEntry (equivalent of PgStatHashEntry)
+ * (native pointer)-> PgStat_Stat*Entry (dsa memory block)
+ *
+ * Local stats that are waiting for being flushed to share stats are stored as:
+ *
+ * pgstat_localhash pgStatLocalHash
+ * -> PgStatLocalHashEntry (local hash entry)
+ * (native pointer)-> PgStat_Stat*Entry/TableStatus (palloc'ed memory)
*/
-#define TABSTAT_QUANTUM 100 /* we alloc this many at a time */
-typedef struct TabStatusArray
+/* The types of statistics entries */
+typedef enum PgStatTypes
{
- struct TabStatusArray *tsa_next; /* link to next array, if any */
- int tsa_used; /* # entries currently used */
- PgStat_TableStatus tsa_entries[TABSTAT_QUANTUM]; /* per-table data */
-} TabStatusArray;
-
-static TabStatusArray *pgStatTabList = NULL;
+ PGSTAT_TYPE_DB, /* database-wide statistics */
+ PGSTAT_TYPE_TABLE, /* per-table statistics */
+ PGSTAT_TYPE_FUNCTION, /* per-function statistics */
+ PGSTAT_TYPE_REPLSLOT /* per-replication-slot statistics */
+} PgStatTypes;
/*
- * pgStatTabHash entry: map from relation OID to PgStat_TableStatus pointer
+ * entry body size lookup table of shared statistics entries corresponding to
+ * PgStatTypes
*/
-typedef struct TabStatHashEntry
+static const size_t pgstat_sharedentsize[] =
{
- Oid t_id;
- PgStat_TableStatus *tsa_entry;
-} TabStatHashEntry;
+ sizeof(PgStat_StatDBEntry), /* PGSTAT_TYPE_DB */
+ sizeof(PgStat_StatTabEntry), /* PGSTAT_TYPE_TABLE */
+ sizeof(PgStat_StatFuncEntry), /* PGSTAT_TYPE_FUNCTION */
+ sizeof(PgStat_ReplSlot) /* PGSTAT_TYPE_REPLSLOT */
+};
-/*
- * Hash table for O(1) t_id -> tsa_entry lookup
- */
-static HTAB *pgStatTabHash = NULL;
+/* Ditto for local statistics entries */
+static const size_t pgstat_localentsize[] =
+{
+ sizeof(PgStat_StatDBEntry), /* PGSTAT_TYPE_DB */
+ sizeof(PgStat_TableStatus), /* PGSTAT_TYPE_TABLE */
+ sizeof(PgStat_BackendFunctionEntry), /* PGSTAT_TYPE_FUNCTION */
+ sizeof(PgStat_ReplSlot) /* PGSTAT_TYPE_REPLSLOT */
+};
/*
- * Backends store per-function info that's waiting to be sent to the collector
- * in this hash table (indexed by function OID).
+ * We shoud avoid overwriting header part of a shared entry. Use these macros
+ * to know what portion of the struct to be written or read. PSTAT_SHENT_BODY
+ * returns a bit smaller address than the actual address of the next member but
+ * that doesn't matter.
*/
-static HTAB *pgStatFunctions = NULL;
+#define PGSTAT_SHENT_BODY(e) (((char *)(e)) + sizeof(PgStat_StatEntryHeader))
+#define PGSTAT_SHENT_BODY_LEN(t) \
+ (pgstat_sharedentsize[t] - sizeof(PgStat_StatEntryHeader))
+
+/* struct for shared statistics hash entry key. */
+typedef struct PgStatHashKey
+{
+ PgStatTypes type; /* statistics entry type */
+ Oid databaseid; /* database ID. InvalidOid for shared objects. */
+ Oid objectid; /* object ID, either table or function. */
+} PgStatHashKey;
+
+/* struct for shared statistics hash entry */
+typedef struct PgStatHashEntry
+{
+ PgStatHashKey key; /* hash key */
+ dsa_pointer body; /* pointer to shared stats in
+ * PgStat_StatEntryHeader */
+} PgStatHashEntry;
+
+/* struct for shared statistics local hash entry. */
+typedef struct PgStatLocalHashEntry
+{
+ PgStatHashKey key; /* hash key */
+ char status; /* for simplehash use */
+ PgStat_StatEntryHeader *body; /* pointer to stats body in local heap */
+ dsa_pointer dsapointer; /* dsa pointer of body */
+} PgStatLocalHashEntry;
+
+/* parameter for the shared hash */
+static const dshash_parameters dsh_params = {
+ sizeof(PgStatHashKey),
+ sizeof(PgStatHashEntry),
+ dshash_memcmp,
+ dshash_memhash,
+ LWTRANCHE_STATS
+};
+
+/* define hashtable for local hashes */
+#define SH_PREFIX pgstat_localhash
+#define SH_ELEMENT_TYPE PgStatLocalHashEntry
+#define SH_KEY_TYPE PgStatHashKey
+#define SH_KEY key
+#define SH_HASH_KEY(tb, key) \
+ hash_bytes((unsigned char *)&key, sizeof(PgStatHashKey))
+#define SH_EQUAL(tb, a, b) (memcmp(&a, &b, sizeof(PgStatHashKey)) == 0)
+#define SH_SCOPE static inline
+#define SH_DEFINE
+#define SH_DECLARE
+#include "lib/simplehash.h"
+
+/* The shared hash to index activity stats entries. */
+static dshash_table *pgStatSharedHash = NULL;
/*
- * Indicates if backend has some function stats that it hasn't yet
- * sent to the collector.
+ * The local cache to index shared stats entries.
+ *
+ * This is a local hash to store native pointers to shared hash
+ * entries. pgStatEntHashAge is copied from StatsShmem->gc_count at creation
+ * and garbage collection.
*/
-static bool have_function_stats = false;
+static pgstat_localhash_hash * pgStatEntHash = NULL;
+static int pgStatEntHashAge = 0; /* cache age of pgStatEntHash */
+
+/* Local stats numbers are stored here. */
+static pgstat_localhash_hash * pgStatLocalHash = NULL;
+
+/* entry type for oid hash */
+typedef struct pgstat_oident
+{
+ Oid oid;
+ char status;
+} pgstat_oident;
+
+/* Define hashtable for OID hashes. */
+StaticAssertDecl(sizeof(Oid) == 4, "oid is not compatible with uint32");
+#define SH_PREFIX pgstat_oid
+#define SH_ELEMENT_TYPE pgstat_oident
+#define SH_KEY_TYPE Oid
+#define SH_KEY oid
+#define SH_HASH_KEY(tb, key) hash_bytes_uint32(key)
+#define SH_EQUAL(tb, a, b) (a == b)
+#define SH_SCOPE static inline
+#define SH_DEFINE
+#define SH_DECLARE
+#include "lib/simplehash.h"
/*
* Tuple insertion/deletion counts for an open transaction can't be propagated
@@ -276,11 +415,8 @@ typedef struct TwoPhasePgStatRecord
bool t_truncated; /* was the relation truncated? */
} TwoPhasePgStatRecord;
-/*
- * Info about current "snapshot" of stats file
- */
+/* Variables for backend status snapshot */
static MemoryContext pgStatLocalContext = NULL;
-static HTAB *pgStatDBHash = NULL;
/* Status for backends including auxiliary */
static LocalPgBackendStatus *localBackendStatusTable = NULL;
@@ -289,23 +425,9 @@ static LocalPgBackendStatus *localBackendStatusTable = NULL;
static int localNumBackends = 0;
/*
- * Cluster wide statistics, kept in the stats collector.
- * Contains statistics that are not collected per database
- * or per table.
+ * Make our own memory context to make it easy to track memory usage.
*/
-static PgStat_ArchiverStats archiverStats;
-static PgStat_GlobalStats globalStats;
-static PgStat_WalStats walStats;
-static PgStat_SLRUStats slruStats[SLRU_NUM_ELEMENTS];
-static PgStat_ReplSlotStats *replSlotStats;
-static int nReplSlotStats;
-
-/*
- * List of OIDs of databases we need to write out. If an entry is InvalidOid,
- * it means to write only the shared-catalog stats ("DB 0"); otherwise, we
- * will write both that DB's data and the shared stats.
- */
-static List *pending_write_requests = NIL;
+MemoryContext pgStatCacheContext = NULL;
/*
* Total time charged to functions so far in the current backend.
@@ -314,41 +436,58 @@ static List *pending_write_requests = NIL;
*/
static instr_time total_func_time;
+/* Simple caching feature for pgstatfuncs */
+static PgStatHashKey stathashkey_zero = {0};
+static PgStatHashKey cached_dbent_key = {0};
+static PgStat_StatDBEntry cached_dbent;
+static PgStatHashKey cached_tabent_key = {0};
+static PgStat_StatTabEntry cached_tabent;
+static PgStatHashKey cached_funcent_key = {0};
+static PgStat_StatFuncEntry cached_funcent;
+
+static PgStat_Archiver cached_archiverstats;
+static bool cached_archiverstats_is_valid = false;
+static PgStat_BgWriter cached_bgwriterstats;
+static bool cached_bgwriterstats_is_valid = false;
+static PgStat_CheckPointer cached_checkpointerstats;
+static bool cached_checkpointerstats_is_valid = false;
+static PgStat_Wal cached_walstats;
+static bool cached_walstats_is_valid = false;
+static PgStat_SLRUStats cached_slrustats;
+static bool cached_slrustats_is_valid = false;
+static PgStat_ReplSlot *cached_replslotstats = NULL;
+static int n_cached_replslotstats = -1;
/* ----------
* Local function forward declarations
* ----------
*/
-#ifdef EXEC_BACKEND
-static pid_t pgstat_forkexec(void);
-#endif
-
-NON_EXEC_STATIC void PgstatCollectorMain(int argc, char *argv[]) pg_attribute_noreturn();
static void pgstat_beshutdown_hook(int code, Datum arg);
-static PgStat_StatDBEntry *pgstat_get_db_entry(Oid databaseid, bool create);
-static PgStat_StatTabEntry *pgstat_get_tab_entry(PgStat_StatDBEntry *dbentry,
- Oid tableoid, bool create);
-static void pgstat_write_statsfiles(bool permanent, bool allDbs);
-static void pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent);
-static HTAB *pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep);
-static void pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash, bool permanent);
-static void backend_read_statsfile(void);
+static PgStat_StatDBEntry *get_local_dbstat_entry(Oid dbid);
+static PgStat_TableStatus *get_local_tabstat_entry(Oid rel_id, bool isshared);
+
+static void pgstat_write_statsfile(void);
+
+static void pgstat_read_statsfile(void);
static void pgstat_read_current_status(void);
-static bool pgstat_write_statsfile_needed(void);
-static bool pgstat_db_requested(Oid databaseid);
+static PgStat_StatEntryHeader *get_stat_entry(PgStatTypes type, Oid dbid,
+ Oid objid, bool nowait,
+ bool create, bool *found);
-static int pgstat_replslot_index(const char *name, bool create_it);
-static void pgstat_reset_replslot(int i, TimestampTz ts);
+static bool flush_tabstat(PgStatLocalHashEntry *ent, bool nowait);
+static bool flush_funcstat(PgStatLocalHashEntry *ent, bool nowait);
+static bool flush_dbstat(PgStatLocalHashEntry *ent, bool nowait);
+static bool flush_walstat(bool nowait);
+static bool flush_slrustat(bool nowait);
+static void delete_current_stats_entry(dshash_seq_status *hstat);
+static PgStat_StatEntryHeader *get_local_stat_entry(PgStatTypes type, Oid dbid,
+ Oid objid, bool create,
+ bool *found);
-static void pgstat_send_tabstat(PgStat_MsgTabstat *tsmsg);
-static void pgstat_send_funcstats(void);
-static void pgstat_send_slru(void);
-static HTAB *pgstat_collect_oids(Oid catalogid, AttrNumber anum_oid);
-static void pgstat_send_connstats(bool disconnect, TimestampTz last_report);
-
-static PgStat_TableStatus *get_tabstat_entry(Oid rel_id, bool isshared);
+static pgstat_oid_hash * collect_oids(Oid catalogid, AttrNumber anum_oid);
+static void pgstat_update_connstats(bool disconnect);
static void pgstat_setup_memcxt(void);
@@ -358,492 +497,651 @@ static const char *pgstat_get_wait_ipc(WaitEventIPC w);
static const char *pgstat_get_wait_timeout(WaitEventTimeout w);
static const char *pgstat_get_wait_io(WaitEventIO w);
-static void pgstat_setheader(PgStat_MsgHdr *hdr, StatMsgType mtype);
-static void pgstat_send(void *msg, int len);
-
-static void pgstat_recv_inquiry(PgStat_MsgInquiry *msg, int len);
-static void pgstat_recv_tabstat(PgStat_MsgTabstat *msg, int len);
-static void pgstat_recv_tabpurge(PgStat_MsgTabpurge *msg, int len);
-static void pgstat_recv_dropdb(PgStat_MsgDropdb *msg, int len);
-static void pgstat_recv_resetcounter(PgStat_MsgResetcounter *msg, int len);
-static void pgstat_recv_resetsharedcounter(PgStat_MsgResetsharedcounter *msg, int len);
-static void pgstat_recv_resetsinglecounter(PgStat_MsgResetsinglecounter *msg, int len);
-static void pgstat_recv_resetslrucounter(PgStat_MsgResetslrucounter *msg, int len);
-static void pgstat_recv_resetreplslotcounter(PgStat_MsgResetreplslotcounter *msg, int len);
-static void pgstat_recv_autovac(PgStat_MsgAutovacStart *msg, int len);
-static void pgstat_recv_vacuum(PgStat_MsgVacuum *msg, int len);
-static void pgstat_recv_analyze(PgStat_MsgAnalyze *msg, int len);
-static void pgstat_recv_archiver(PgStat_MsgArchiver *msg, int len);
-static void pgstat_recv_bgwriter(PgStat_MsgBgWriter *msg, int len);
-static void pgstat_recv_wal(PgStat_MsgWal *msg, int len);
-static void pgstat_recv_slru(PgStat_MsgSLRU *msg, int len);
-static void pgstat_recv_funcstat(PgStat_MsgFuncstat *msg, int len);
-static void pgstat_recv_funcpurge(PgStat_MsgFuncpurge *msg, int len);
-static void pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int len);
-static void pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len);
-static void pgstat_recv_checksum_failure(PgStat_MsgChecksumFailure *msg, int len);
-static void pgstat_recv_connstat(PgStat_MsgConn *msg, int len);
-static void pgstat_recv_replslot(PgStat_MsgReplSlot *msg, int len);
-static void pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len);
-
/* ------------------------------------------------------------
* Public functions called from postmaster follow
* ------------------------------------------------------------
*/
-/* ----------
- * pgstat_init() -
- *
- * Called from postmaster at startup. Create the resources required
- * by the statistics collector process. If unable to do so, do not
- * fail --- better to let the postmaster start with stats collection
- * disabled.
- * ----------
+/*
+ * StatsShmemSize
+ * Compute shared memory space needed for activity statistic
+ */
+Size
+StatsShmemSize(void)
+{
+ return sizeof(StatsShmemStruct);
+}
+
+/*
+ * StatsShmemInit - initialize during shared-memory creation
*/
void
-pgstat_init(void)
+StatsShmemInit(void)
{
- ACCEPT_TYPE_ARG3 alen;
- struct addrinfo *addrs = NULL,
- *addr,
- hints;
- int ret;
- fd_set rset;
- struct timeval tv;
- char test_byte;
- int sel_res;
- int tries = 0;
-
-#define TESTBYTEVAL ((char) 199)
+ bool found;
- /*
- * This static assertion verifies that we didn't mess up the calculations
- * involved in selecting maximum payload sizes for our UDP messages.
- * Because the only consequence of overrunning PGSTAT_MAX_MSG_SIZE would
- * be silent performance loss from fragmentation, it seems worth having a
- * compile-time cross-check that we didn't.
- */
- StaticAssertStmt(sizeof(PgStat_Msg) <= PGSTAT_MAX_MSG_SIZE,
- "maximum stats message size exceeds PGSTAT_MAX_MSG_SIZE");
+ StatsShmem = (StatsShmemStruct *)
+ ShmemInitStruct("Stats area", StatsShmemSize(), &found);
- /*
- * Create the UDP socket for sending and receiving statistic messages
- */
- hints.ai_flags = AI_PASSIVE;
- hints.ai_family = AF_UNSPEC;
- hints.ai_socktype = SOCK_DGRAM;
- hints.ai_protocol = 0;
- hints.ai_addrlen = 0;
- hints.ai_addr = NULL;
- hints.ai_canonname = NULL;
- hints.ai_next = NULL;
- ret = pg_getaddrinfo_all("localhost", NULL, &hints, &addrs);
- if (ret || !addrs)
+ if (!IsUnderPostmaster)
+ {
+ Assert(!found);
+
+ StatsShmem->stats_dsa_handle = DSM_HANDLE_INVALID;
+ ConditionVariableInit(&StatsShmem->holdoff_cv);
+ pg_atomic_init_u32(&StatsShmem->archiver_changecount, 0);
+ pg_atomic_init_u32(&StatsShmem->bgwriter_changecount, 0);
+ pg_atomic_init_u32(&StatsShmem->checkpointer_changecount, 0);
+
+ pg_atomic_init_u64(&StatsShmem->gc_count, 0);
+
+ LWLockInitialize(&StatsShmem->wal_stats_lock, LWTRANCHE_STATS);
+ }
+}
+
+/* ----------
+ * allow_next_attacher() -
+ *
+ * Let other processes to go ahead attaching the shared stats area.
+ * ----------
+ */
+static void
+allow_next_attacher(void)
+{
+ bool triggerd = false;
+
+ LWLockAcquire(StatsLock, LW_EXCLUSIVE);
+ if (StatsShmem->attach_holdoff)
{
- ereport(LOG,
- (errmsg("could not resolve \"localhost\": %s",
- gai_strerror(ret))));
- goto startup_failed;
+ StatsShmem->attach_holdoff = false;
+ triggerd = true;
}
+ LWLockRelease(StatsLock);
+
+ if (triggerd)
+ ConditionVariableBroadcast(&StatsShmem->holdoff_cv);
+}
+
+/* ----------
+ * attach_shared_stats() -
+ *
+ * Attach shared or create stats memory. If we are the first process to use
+ * activity stats system, read the saved statistics file if any.
+ * ---------
+ */
+static void
+attach_shared_stats(void)
+{
+ MemoryContext oldcontext;
/*
- * On some platforms, pg_getaddrinfo_all() may return multiple addresses
- * only one of which will actually work (eg, both IPv6 and IPv4 addresses
- * when kernel will reject IPv6). Worse, the failure may occur at the
- * bind() or perhaps even connect() stage. So we must loop through the
- * results till we find a working combination. We will generate LOG
- * messages, but no error, for bogus combinations.
+ * Don't use dsm under postmaster, or when not tracking counts.
*/
- for (addr = addrs; addr; addr = addr->ai_next)
- {
-#ifdef HAVE_UNIX_SOCKETS
- /* Ignore AF_UNIX sockets, if any are returned. */
- if (addr->ai_family == AF_UNIX)
- continue;
-#endif
+ if (!pgstat_track_counts || !IsUnderPostmaster)
+ return;
- if (++tries > 1)
- ereport(LOG,
- (errmsg("trying another address for the statistics collector")));
+ pgstat_setup_memcxt();
- /*
- * Create the socket.
- */
- if ((pgStatSock = socket(addr->ai_family, SOCK_DGRAM, 0)) == PGINVALID_SOCKET)
- {
- ereport(LOG,
- (errcode_for_socket_access(),
- errmsg("could not create socket for statistics collector: %m")));
- continue;
- }
+ if (area)
+ return;
- /*
- * Bind it to a kernel assigned port on localhost and get the assigned
- * port via getsockname().
- */
- if (bind(pgStatSock, addr->ai_addr, addr->ai_addrlen) < 0)
- {
- ereport(LOG,
- (errcode_for_socket_access(),
- errmsg("could not bind socket for statistics collector: %m")));
- closesocket(pgStatSock);
- pgStatSock = PGINVALID_SOCKET;
- continue;
- }
-
- alen = sizeof(pgStatAddr);
- if (getsockname(pgStatSock, (struct sockaddr *) &pgStatAddr, &alen) < 0)
- {
- ereport(LOG,
- (errcode_for_socket_access(),
- errmsg("could not get address of socket for statistics collector: %m")));
- closesocket(pgStatSock);
- pgStatSock = PGINVALID_SOCKET;
- continue;
- }
+ /* stats shared memory persists for the backend lifetime */
+ oldcontext = MemoryContextSwitchTo(TopMemoryContext);
- /*
- * Connect the socket to its own address. This saves a few cycles by
- * not having to respecify the target address on every send. This also
- * provides a kernel-level check that only packets from this same
- * address will be received.
- */
- if (connect(pgStatSock, (struct sockaddr *) &pgStatAddr, alen) < 0)
- {
- ereport(LOG,
- (errcode_for_socket_access(),
- errmsg("could not connect socket for statistics collector: %m")));
- closesocket(pgStatSock);
- pgStatSock = PGINVALID_SOCKET;
- continue;
- }
+ /*
+ * The first attacher backend may still reading the stats file, or the
+ * last detacher may writing it. Wait for the work to finish.
+ */
+ ConditionVariablePrepareToSleep(&StatsShmem->holdoff_cv);
+ for (;;)
+ {
+ bool hold_off;
- /*
- * Try to send and receive a one-byte test message on the socket. This
- * is to catch situations where the socket can be created but will not
- * actually pass data (for instance, because kernel packet filtering
- * rules prevent it).
- */
- test_byte = TESTBYTEVAL;
-
-retry1:
- if (send(pgStatSock, &test_byte, 1, 0) != 1)
- {
- if (errno == EINTR)
- goto retry1; /* if interrupted, just retry */
- ereport(LOG,
- (errcode_for_socket_access(),
- errmsg("could not send test message on socket for statistics collector: %m")));
- closesocket(pgStatSock);
- pgStatSock = PGINVALID_SOCKET;
- continue;
- }
+ LWLockAcquire(StatsLock, LW_SHARED);
+ hold_off = StatsShmem->attach_holdoff;
+ LWLockRelease(StatsLock);
- /*
- * There could possibly be a little delay before the message can be
- * received. We arbitrarily allow up to half a second before deciding
- * it's broken.
- */
- for (;;) /* need a loop to handle EINTR */
- {
- FD_ZERO(&rset);
- FD_SET(pgStatSock, &rset);
-
- tv.tv_sec = 0;
- tv.tv_usec = 500000;
- sel_res = select(pgStatSock + 1, &rset, NULL, NULL, &tv);
- if (sel_res >= 0 || errno != EINTR)
- break;
- }
- if (sel_res < 0)
- {
- ereport(LOG,
- (errcode_for_socket_access(),
- errmsg("select() failed in statistics collector: %m")));
- closesocket(pgStatSock);
- pgStatSock = PGINVALID_SOCKET;
- continue;
- }
- if (sel_res == 0 || !FD_ISSET(pgStatSock, &rset))
- {
- /*
- * This is the case we actually think is likely, so take pains to
- * give a specific message for it.
- *
- * errno will not be set meaningfully here, so don't use it.
- */
- ereport(LOG,
- (errcode(ERRCODE_CONNECTION_FAILURE),
- errmsg("test message did not get through on socket for statistics collector")));
- closesocket(pgStatSock);
- pgStatSock = PGINVALID_SOCKET;
- continue;
- }
-
- test_byte++; /* just make sure variable is changed */
-
-retry2:
- if (recv(pgStatSock, &test_byte, 1, 0) != 1)
- {
- if (errno == EINTR)
- goto retry2; /* if interrupted, just retry */
- ereport(LOG,
- (errcode_for_socket_access(),
- errmsg("could not receive test message on socket for statistics collector: %m")));
- closesocket(pgStatSock);
- pgStatSock = PGINVALID_SOCKET;
- continue;
- }
-
- if (test_byte != TESTBYTEVAL) /* strictly paranoia ... */
- {
- ereport(LOG,
- (errcode(ERRCODE_INTERNAL_ERROR),
- errmsg("incorrect test message transmission on socket for statistics collector")));
- closesocket(pgStatSock);
- pgStatSock = PGINVALID_SOCKET;
- continue;
- }
-
- /* If we get here, we have a working socket */
- break;
+ if (!hold_off)
+ break;
+
+ ConditionVariableTimedSleep(&StatsShmem->holdoff_cv, 10,
+ WAIT_EVENT_READING_STATS_FILE);
}
+ ConditionVariableCancelSleep();
- /* Did we find a working address? */
- if (!addr || pgStatSock == PGINVALID_SOCKET)
- goto startup_failed;
+ LWLockAcquire(StatsLock, LW_EXCLUSIVE);
/*
- * Set the socket to non-blocking IO. This ensures that if the collector
- * falls behind, statistics messages will be discarded; backends won't
- * block waiting to send messages to the collector.
+ * The last process is responsible to write out stats files at exit.
+ * Maintain refcount so that a process going to exit can find whether it
+ * is the last one or not.
*/
- if (!pg_set_noblock(pgStatSock))
+ if (StatsShmem->refcount > 0)
+ StatsShmem->refcount++;
+ else
{
- ereport(LOG,
- (errcode_for_socket_access(),
- errmsg("could not set statistics collector socket to nonblocking mode: %m")));
- goto startup_failed;
+ /* We're the first process to attach the shared stats memory */
+ Assert(StatsShmem->stats_dsa_handle == DSM_HANDLE_INVALID);
+
+ /* Initialize shared memory area */
+ area = dsa_create(LWTRANCHE_STATS);
+ pgStatSharedHash = dshash_create(area, &dsh_params, 0);
+
+ StatsShmem->stats_dsa_handle = dsa_get_handle(area);
+ StatsShmem->hash_handle = dshash_get_hash_table_handle(pgStatSharedHash);
+ LWLockInitialize(&StatsShmem->slru_stats.lock, LWTRANCHE_STATS);
+ pg_atomic_init_u32(&StatsShmem->slru_stats.changecount, 0);
+
+ /* Block the next attacher for a while, see the comment above. */
+ StatsShmem->attach_holdoff = true;
+
+ StatsShmem->refcount = 1;
}
- /*
- * Try to ensure that the socket's receive buffer is at least
- * PGSTAT_MIN_RCVBUF bytes, so that it won't easily overflow and lose
- * data. Use of UDP protocol means that we are willing to lose data under
- * heavy load, but we don't want it to happen just because of ridiculously
- * small default buffer sizes (such as 8KB on older Windows versions).
- */
+ LWLockRelease(StatsLock);
+
+ if (area)
{
- int old_rcvbuf;
- int new_rcvbuf;
- ACCEPT_TYPE_ARG3 rcvbufsize = sizeof(old_rcvbuf);
-
- if (getsockopt(pgStatSock, SOL_SOCKET, SO_RCVBUF,
- (char *) &old_rcvbuf, &rcvbufsize) < 0)
- {
- ereport(LOG,
- (errmsg("getsockopt(%s) failed: %m", "SO_RCVBUF")));
- /* if we can't get existing size, always try to set it */
- old_rcvbuf = 0;
- }
-
- new_rcvbuf = PGSTAT_MIN_RCVBUF;
- if (old_rcvbuf < new_rcvbuf)
- {
- if (setsockopt(pgStatSock, SOL_SOCKET, SO_RCVBUF,
- (char *) &new_rcvbuf, sizeof(new_rcvbuf)) < 0)
- ereport(LOG,
- (errmsg("setsockopt(%s) failed: %m", "SO_RCVBUF")));
- }
+ /*
+ * We're the first attacher process, read stats file while blocking
+ * successors.
+ */
+ Assert(StatsShmem->attach_holdoff);
+ pgstat_read_statsfile();
+ allow_next_attacher();
+ }
+ else
+ {
+ /* We're not the first one, attach existing shared area. */
+ area = dsa_attach(StatsShmem->stats_dsa_handle);
+ pgStatSharedHash = dshash_attach(area, &dsh_params,
+ StatsShmem->hash_handle, 0);
}
- pg_freeaddrinfo_all(hints.ai_family, addrs);
+ Assert(StatsShmem->stats_dsa_handle != DSM_HANDLE_INVALID);
- /* Now that we have a long-lived socket, tell fd.c about it. */
- ReserveExternalFD();
+ MemoryContextSwitchTo(oldcontext);
- return;
+ /* don't detach automatically */
+ dsa_pin_mapping(area);
+}
-startup_failed:
- ereport(LOG,
- (errmsg("disabling statistics collector for lack of working socket")));
+/* ----------
+ * cleanup_dropped_stats_entries() -
+ * Clean up shared stats entries no longer used.
+ *
+ * Shared stats entries for dropped objects may be left referenced. Clean up
+ * our reference and drop the shared entry if needed.
+ * ----------
+ */
+static void
+cleanup_dropped_stats_entries(void)
+{
+ pgstat_localhash_iterator i;
+ PgStatLocalHashEntry *ent;
- if (addrs)
- pg_freeaddrinfo_all(hints.ai_family, addrs);
+ if (pgStatEntHash == NULL)
+ return;
- if (pgStatSock != PGINVALID_SOCKET)
- closesocket(pgStatSock);
- pgStatSock = PGINVALID_SOCKET;
+ pgstat_localhash_start_iterate(pgStatEntHash, &i);
+ while ((ent = pgstat_localhash_iterate(pgStatEntHash, &i))
+ != NULL)
+ {
+ /*
+ * Free the shared memory chunk for the entry if we were the last
+ * referrer to a dropped entry.
+ */
+ if (pg_atomic_sub_fetch_u32(&ent->body->refcount, 1) < 1 &&
+ ent->body->dropped)
+ dsa_free(area, ent->dsapointer);
+ }
/*
- * Adjust GUC variables to suppress useless activity, and for debugging
- * purposes (seeing track_counts off is a clue that we failed here). We
- * use PGC_S_OVERRIDE because there is no point in trying to turn it back
- * on from postgresql.conf without a restart.
+ * This function is expected to be called during backend exit. So we don't
+ * bother destroying pgStatEntHash.
*/
- SetConfigOption("track_counts", "off", PGC_INTERNAL, PGC_S_OVERRIDE);
+ pgStatEntHash = NULL;
}
-/*
- * subroutine for pgstat_reset_all
+/* ----------
+ * detach_shared_stats() -
+ *
+ * Detach shared stats. Write out to file if we're the last process and told
+ * to do so.
+ * ----------
*/
static void
-pgstat_reset_remove_files(const char *directory)
+detach_shared_stats(bool write_file)
{
- DIR *dir;
- struct dirent *entry;
- char fname[MAXPGPATH * 2];
+ bool is_last_detacher = 0;
+
+ /* immediately return if useless */
+ if (!area || !IsUnderPostmaster)
+ return;
+
+ /* We shouldn't leave a reference to shared stats. */
+ cleanup_dropped_stats_entries();
- dir = AllocateDir(directory);
- while ((entry = ReadDir(dir, directory)) != NULL)
+ /*
+ * If we are the last detacher, hold off the next attacher (if possible)
+ * until we finish writing stats file.
+ */
+ LWLockAcquire(StatsLock, LW_EXCLUSIVE);
+ if (--StatsShmem->refcount == 0)
{
- int nchars;
- Oid tmp_oid;
+ StatsShmem->attach_holdoff = true;
+ is_last_detacher = true;
+ }
+ LWLockRelease(StatsLock);
- /*
- * Skip directory entries that don't match the file names we write.
- * See get_dbstat_filename for the database-specific pattern.
- */
- if (strncmp(entry->d_name, "global.", 7) == 0)
- nchars = 7;
- else
- {
- nchars = 0;
- (void) sscanf(entry->d_name, "db_%u.%n",
- &tmp_oid, &nchars);
- if (nchars <= 0)
- continue;
- /* %u allows leading whitespace, so reject that */
- if (strchr("0123456789", entry->d_name[3]) == NULL)
- continue;
- }
-
- if (strcmp(entry->d_name + nchars, "tmp") != 0 &&
- strcmp(entry->d_name + nchars, "stat") != 0)
- continue;
-
- snprintf(fname, sizeof(fname), "%s/%s", directory,
- entry->d_name);
- unlink(fname);
+ if (is_last_detacher)
+ {
+ if (write_file)
+ pgstat_write_statsfile();
+
+ StatsShmem->stats_dsa_handle = DSM_HANDLE_INVALID;
+ /* allow the next attacher, if any */
+ allow_next_attacher();
}
- FreeDir(dir);
+
+ /*
+ * Detach the area. It is automatically destroyed when the last process
+ * detached it.
+ */
+ dsa_detach(area);
+
+ area = NULL;
+ pgStatSharedHash = NULL;
+
+ /* We are going to exit. Don't bother destroying local hashes. */
+ pgStatLocalHash = NULL;
}
/*
* pgstat_reset_all() -
*
- * Remove the stats files. This is currently used only if WAL
- * recovery is needed after a crash.
+ * Remove the stats file. This is currently used only if WAL recovery is
+ * needed after a crash.
*/
void
pgstat_reset_all(void)
{
- pgstat_reset_remove_files(pgstat_stat_directory);
- pgstat_reset_remove_files(PGSTAT_STAT_PERMANENT_DIRECTORY);
+ /* standalone server doesn't use shared stats */
+ if (!IsUnderPostmaster)
+ return;
+
+ /* we must have shared stats attached */
+ Assert(StatsShmem->stats_dsa_handle != DSM_HANDLE_INVALID);
+
+ /* Startup must be the only user of shared stats */
+ Assert(StatsShmem->refcount == 1);
+
+ /*
+ * We could directly remove files and recreate the shared memory area. But
+ * just discard then create for simplicity.
+ */
+ detach_shared_stats(false);
+ attach_shared_stats();
}
-#ifdef EXEC_BACKEND
/*
- * pgstat_forkexec() -
- *
- * Format up the arglist for, then fork and exec, statistics collector process
+ * fetch_lock_statentry - common helper function to fetch and lock a stats
+ * entry for flush_tabstat, flush_funcstat and flush_dbstat.
*/
-static pid_t
-pgstat_forkexec(void)
+static PgStat_StatEntryHeader *
+fetch_lock_statentry(PgStatTypes type, Oid dboid, Oid objid, bool nowait)
{
- char *av[10];
- int ac = 0;
+ PgStat_StatEntryHeader *header;
+
+ /* find shared table stats entry corresponding to the local entry */
+ header = (PgStat_StatEntryHeader *)
+ get_stat_entry(type, dboid, objid, nowait, true, NULL);
- av[ac++] = "postgres";
- av[ac++] = "--forkcol";
- av[ac++] = NULL; /* filled in by postmaster_forkexec */
+ /* skip if dshash failed to acquire lock */
+ if (header == NULL)
+ return false;
- av[ac] = NULL;
- Assert(ac < lengthof(av));
+ /* lock the shared entry to protect the content, skip if failed */
+ if (!nowait)
+ LWLockAcquire(&header->lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(&header->lock, LW_EXCLUSIVE))
+ return false;
- return postmaster_forkexec(ac, av);
+ return header;
}
-#endif /* EXEC_BACKEND */
-/*
- * pgstat_start() -
+/* ----------
+ * get_stat_entry() -
*
- * Called from postmaster at startup or after an existing collector
- * died. Attempt to fire up a fresh statistics collector.
+ * get shared stats entry for specified type, dbid and objid.
+ * If nowait is true, returns NULL on lock failure.
*
- * Returns PID of child process, or 0 if fail.
+ * If initfunc is not NULL, new entry is created if not yet and the function
+ * is called with the new base entry. If found is not NULL, it is set to true
+ * if existing entry is found or false if not.
+ * ----------
+ */
+static PgStat_StatEntryHeader *
+get_stat_entry(PgStatTypes type, Oid dbid, Oid objid, bool nowait, bool create,
+ bool *found)
+{
+ PgStatHashEntry *shhashent;
+ PgStatLocalHashEntry *lohashent;
+ PgStat_StatEntryHeader *shheader = NULL;
+ PgStatHashKey key;
+ bool shfound;
+
+ key.type = type;
+ key.databaseid = dbid;
+ key.objectid = objid;
+
+ if (pgStatEntHash)
+ {
+ uint64 currage;
+
+ /*
+ * pgStatEntHashAge increments quite slowly than the time the
+ * following loop takes so this is expected to iterate no more than
+ * twice.
+ */
+ while (unlikely
+ (pgStatEntHashAge !=
+ (currage = pg_atomic_read_u64(&StatsShmem->gc_count))))
+ {
+ pgstat_localhash_iterator i;
+
+ /*
+ * Some entries have been dropped. Invalidate cache pointer to
+ * them.
+ */
+ pgstat_localhash_start_iterate(pgStatEntHash, &i);
+ while ((lohashent = pgstat_localhash_iterate(pgStatEntHash, &i))
+ != NULL)
+ {
+ PgStat_StatEntryHeader *header = lohashent->body;
+
+ if (header->dropped)
+ {
+ pgstat_localhash_delete(pgStatEntHash, key);
+
+ if (pg_atomic_sub_fetch_u32(&header->refcount, 1) < 1)
+ {
+ /*
+ * We're the last referrer to this entry, drop the
+ * shared entry.
+ */
+ dsa_free(area, lohashent->dsapointer);
+ }
+ }
+ }
+
+ pgStatEntHashAge = currage;
+ }
+
+ lohashent = pgstat_localhash_lookup(pgStatEntHash, key);
+
+ if (lohashent)
+ {
+ if (found)
+ *found = true;
+ return lohashent->body;
+ }
+ }
+
+ shhashent = dshash_find_extended(pgStatSharedHash, &key,
+ create, nowait, create, &shfound);
+ if (shhashent)
+ {
+ if (create && !shfound)
+ {
+ /* Create new stats entry. */
+ dsa_pointer chunk = dsa_allocate0(area,
+ pgstat_sharedentsize[type]);
+
+ shheader = dsa_get_address(area, chunk);
+ LWLockInitialize(&shheader->lock, LWTRANCHE_STATS);
+ pg_atomic_init_u32(&shheader->refcount, 0);
+
+ /* Link the new entry from the hash entry. */
+ shhashent->body = chunk;
+ }
+ else
+ shheader = dsa_get_address(area, shhashent->body);
+
+ /*
+ * We expose this shared entry now. You might think that the entry
+ * can be removed by a concurrent backend, but since we are creating
+ * an stats entry, the object actually exists and used in the upper
+ * layer. Such an object cannot be dropped until the first vacuum
+ * after the current transaction ends.
+ */
+ dshash_release_lock(pgStatSharedHash, shhashent);
+
+ /* register to local hash if possible */
+ if (pgStatEntHash || pgStatCacheContext)
+ {
+ bool lofound;
+
+ if (pgStatEntHash == NULL)
+ {
+ pgStatEntHash =
+ pgstat_localhash_create(pgStatCacheContext,
+ PGSTAT_TABLE_HASH_SIZE, NULL);
+ pgStatEntHashAge =
+ pg_atomic_read_u64(&StatsShmem->gc_count);
+ }
+
+ lohashent =
+ pgstat_localhash_insert(pgStatEntHash, key, &lofound);
+
+ Assert(!lofound);
+ lohashent->body = shheader;
+ lohashent->dsapointer = shhashent->body;
+
+ pg_atomic_add_fetch_u32(&shheader->refcount, 1);
+ }
+ }
+
+ if (found)
+ *found = shfound;
+
+ return shheader;
+}
+
+/*
+ * flush_walstat - flush out a local WAL stats entries
*
- * Note: if fail, we will be called again from the postmaster main loop.
+ * If nowait is true, this function returns false on lock failure. Otherwise
+ * this function always returns true.
+ *
+ * Returns true if all local WAL stats are successfully flushed out.
*/
-int
-pgstat_start(void)
+static bool
+flush_walstat(bool nowait)
{
- time_t curtime;
- pid_t pgStatPid;
+ PgStat_Wal *s = &StatsShmem->wal_stats;
+ PgStat_Wal *l = &WalStats;
+ WalUsage all_zeroes = {0} PG_USED_FOR_ASSERTS_ONLY;
/*
- * Check that the socket is there, else pgstat_init failed and we can do
- * nothing useful.
+ * We don't update the WAL usage portion of the local WalStats elsewhere.
+ * Instead, fill in that portion with the difference of pgWalUsage since
+ * the previous call.
*/
- if (pgStatSock == PGINVALID_SOCKET)
- return 0;
+ Assert(memcmp(&l->wal_usage, &all_zeroes, sizeof(WalUsage)) == 0);
+ WalUsageAccumDiff(&l->wal_usage, &pgWalUsage, &prevWalUsage);
/*
- * Do nothing if too soon since last collector start. This is a safety
- * valve to protect against continuous respawn attempts if the collector
- * is dying immediately at launch. Note that since we will be re-called
- * from the postmaster main loop, we will get another chance later.
+ * This function can be called even if nothing at all has happened. Avoid
+ * taking lock for nothing in that case.
*/
- curtime = time(NULL);
- if ((unsigned int) (curtime - last_pgstat_start_time) <
- (unsigned int) PGSTAT_RESTART_INTERVAL)
- return 0;
- last_pgstat_start_time = curtime;
+ if (!walstats_pending())
+ return true;
+
+ /* lock the shared entry to protect the content, skip if failed */
+ if (!nowait)
+ LWLockAcquire(&StatsShmem->wal_stats_lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(&StatsShmem->wal_stats_lock,
+ LW_EXCLUSIVE))
+ return false; /* failed to acquire lock, skip */
+
+ s->wal_usage.wal_records += l->wal_usage.wal_records;
+ s->wal_usage.wal_fpi += l->wal_usage.wal_fpi;
+ s->wal_usage.wal_bytes += l->wal_usage.wal_bytes;
+ s->wal_buffers_full += l->wal_buffers_full;
+ s->wal_write += l->wal_write;
+ s->wal_write_time += l->wal_write_time;
+ s->wal_sync += l->wal_sync;
+ s->wal_sync_time += l->wal_sync_time;
+ LWLockRelease(&StatsShmem->wal_stats_lock);
/*
- * Okay, fork off the collector.
+ * Save the current counters for the subsequent calculation of WAL usage.
*/
-#ifdef EXEC_BACKEND
- switch ((pgStatPid = pgstat_forkexec()))
-#else
- switch ((pgStatPid = fork_process()))
-#endif
- {
- case -1:
- ereport(LOG,
- (errmsg("could not fork statistics collector: %m")));
- return 0;
+ prevWalUsage = pgWalUsage;
-#ifndef EXEC_BACKEND
- case 0:
- /* in postmaster child ... */
- InitPostmasterChild();
+ /*
+ * Clear out the statistics buffer, so it can be re-used.
+ */
+ MemSet(&WalStats, 0, sizeof(WalStats));
- /* Close the postmaster's sockets */
- ClosePostmasterPorts(false);
+ return true;
+}
- /* Drop our connection to postmaster's shared memory, as well */
- dsm_detach_all();
- PGSharedMemoryDetach();
+/*
+ * flush_slrustat - flush out a local SLRU stats entries
+ *
+ * If nowait is true, this function returns false on lock failure. Otherwise
+ * this function always returns true. Writer processes are mutually excluded
+ * using LWLock, but readers are expected to use change-count protocol to avoid
+ * interference with writers.
+ *
+ * Returns true if all local SLRU stats are successfully flushed out.
+ */
+static bool
+flush_slrustat(bool nowait)
+{
+ uint32 assert_changecount PG_USED_FOR_ASSERTS_ONLY;
+ int i;
- PgstatCollectorMain(0, NULL);
- break;
-#endif
+ if (!have_slrustats)
+ return true;
- default:
- return (int) pgStatPid;
+ /* lock the shared entry to protect the content, skip if failed */
+ if (!nowait)
+ LWLockAcquire(&StatsShmem->slru_stats.lock, LW_EXCLUSIVE);
+ else if (!LWLockConditionalAcquire(&StatsShmem->slru_stats.lock,
+ LW_EXCLUSIVE))
+ return false; /* failed to acquire lock, skip */
+
+ assert_changecount =
+ pg_atomic_fetch_add_u32(&StatsShmem->slru_stats.changecount, 1);
+ Assert((assert_changecount & 1) == 0);
+
+ for (i = 0; i < SLRU_NUM_ELEMENTS; i++)
+ {
+ PgStat_SLRUStats *sharedent = &StatsShmem->slru_stats.entry[i];
+ PgStat_SLRUStats *localent = &local_SLRUStats[i];
+
+ sharedent->blocks_zeroed += localent->blocks_zeroed;
+ sharedent->blocks_hit += localent->blocks_hit;
+ sharedent->blocks_read += localent->blocks_read;
+ sharedent->blocks_written += localent->blocks_written;
+ sharedent->blocks_exists += localent->blocks_exists;
+ sharedent->flush += localent->flush;
+ sharedent->truncate += localent->truncate;
}
- /* shouldn't get here */
- return 0;
+ /* done, clear the local entry */
+ MemSet(local_SLRUStats, 0,
+ sizeof(PgStat_SLRUStats) * SLRU_NUM_ELEMENTS);
+
+ pg_atomic_add_fetch_u32(&StatsShmem->slru_stats.changecount, 1);
+ LWLockRelease(&StatsShmem->slru_stats.lock);
+
+ have_slrustats = false;
+
+ return true;
+}
+
+/* ----------
+ * delete_current_stats_entry()
+ *
+ * Deletes the given shared entry from shared stats hash. The entry must be
+ * exclusively locked.
+ * ----------
+ */
+static void
+delete_current_stats_entry(dshash_seq_status *hstat)
+{
+ dsa_pointer pdsa;
+ PgStat_StatEntryHeader *header;
+ PgStatHashEntry *ent;
+
+ ent = dshash_get_current(hstat);
+ pdsa = ent->body;
+ header = dsa_get_address(area, pdsa);
+
+ /* No one find this entry ever after. */
+ dshash_delete_current(hstat);
+
+ /*
+ * Let the referrers drop the entry if any. Refcount won't be decremented
+ * until "dropped" is set true and StatsShmem->gc_count is incremented
+ * later. So we can check refcount to set dropped without holding a lock.
+ * If no one is referring this entry, free it immediately.
+ */
+
+ if (pg_atomic_read_u32(&header->refcount) > 0)
+ header->dropped = true;
+ else
+ dsa_free(area, pdsa);
+
+ return;
}
-void
-allow_immediate_pgstat_restart(void)
+/* ----------
+ * get_local_stat_entry() -
+ *
+ * Returns local stats entry for the type, dbid and objid.
+ * If create is true, new entry is created if not yet. found must be non-null
+ * in the case.
+ *
+ *
+ * The caller is responsible to initialize the statsbody part of the returned
+ * memory.
+ * ----------
+ */
+static PgStat_StatEntryHeader *
+get_local_stat_entry(PgStatTypes type, Oid dbid, Oid objid,
+ bool create, bool *found)
{
- last_pgstat_start_time = 0;
+ PgStatHashKey key;
+ PgStatLocalHashEntry *entry;
+
+ if (pgStatLocalHash == NULL)
+ pgStatLocalHash = pgstat_localhash_create(pgStatCacheContext,
+ PGSTAT_TABLE_HASH_SIZE, NULL);
+
+ /* Find an entry or create a new one. */
+ key.type = type;
+ key.databaseid = dbid;
+ key.objectid = objid;
+ if (create)
+ entry = pgstat_localhash_insert(pgStatLocalHash, key, found);
+ else
+ entry = pgstat_localhash_lookup(pgStatLocalHash, key);
+
+ if (!create && !entry)
+ return NULL;
+
+ if (create && !*found)
+ entry->body = MemoryContextAllocZero(TopMemoryContext,
+ pgstat_localentsize[type]);
+
+ return entry->body;
}
/* ------------------------------------------------------------
@@ -851,159 +1149,415 @@ allow_immediate_pgstat_restart(void)
*------------------------------------------------------------
*/
-
/* ----------
* pgstat_report_stat() -
*
* Must be called by processes that performs DML: tcop/postgres.c, logical
- * receiver processes, SPI worker, etc. to send the so far collected
- * per-table and function usage statistics to the collector. Note that this
- * is called only when not within a transaction, so it is fair to use
+ * receiver processes, SPI worker, etc. to apply the so far collected
+ * per-table and function usage statistics to the shared statistics hashes.
+ *
+ * Updates are applied not more frequent than the interval of
+ * PGSTAT_MIN_INTERVAL milliseconds. They are also postponed on lock
+ * failure if force is false and there's no pending updates longer than
+ * PGSTAT_MAX_INTERVAL milliseconds. Postponed updates are retried in
+ * succeeding calls of this function.
+ *
+ * Returns the time until the next timing when updates are applied in
+ * milliseconds if there are no updates held for more than
+ * PGSTAT_MIN_INTERVAL milliseconds.
+ *
+ * Note that this is called only out of a transaction, so it is fine to use
* transaction stop time as an approximation of current time.
- *
- * "disconnect" is "true" only for the last call before the backend
- * exits. This makes sure that no data is lost and that interrupted
- * sessions are reported correctly.
- * ----------
+ * ----------
*/
-void
-pgstat_report_stat(bool disconnect)
+long
+pgstat_report_stat(bool force)
{
- /* we assume this inits to all zeroes: */
- static const PgStat_TableCounts all_zeroes;
- static TimestampTz last_report = 0;
-
+ static TimestampTz next_flush = 0;
+ static TimestampTz pending_since = 0;
+ static long retry_interval = 0;
TimestampTz now;
- PgStat_MsgTabstat regular_msg;
- PgStat_MsgTabstat shared_msg;
- TabStatusArray *tsa;
+ bool nowait;
int i;
+ uint64 oldval;
+
+ /* Return if not active */
+ if (area == NULL)
+ return 0;
+
+ /*
+ * We need a database entry if the following stats exists.
+ */
+ if (pgStatXactCommit > 0 || pgStatXactRollback > 0 ||
+ pgStatBlockReadTime > 0 || pgStatBlockWriteTime > 0)
+ get_local_dbstat_entry(MyDatabaseId);
/* Don't expend a clock check if nothing to do */
- if ((pgStatTabList == NULL || pgStatTabList->tsa_used == 0) &&
- pgStatXactCommit == 0 && pgStatXactRollback == 0 &&
- !have_function_stats && !disconnect)
- return;
+ if (pgStatLocalHash == NULL && have_slrustats && !walstats_pending())
+ return 0;
- /*
- * Don't send a message unless it's been at least PGSTAT_STAT_INTERVAL
- * msec since we last sent one, or the backend is about to exit.
- */
now = GetCurrentTransactionStopTimestamp();
- if (!disconnect &&
- !TimestampDifferenceExceeds(last_report, now, PGSTAT_STAT_INTERVAL))
- return;
- /* for backends, send connection statistics */
+ if (!force)
+ {
+ /*
+ * Don't flush stats too frequently. Return the time to the next
+ * flush.
+ */
+ if (now < next_flush)
+ {
+ /* Record the epoch time if retrying. */
+ if (pending_since == 0)
+ pending_since = now;
+
+ return (next_flush - now) / 1000;
+ }
+
+ /* But, don't keep pending updates longer than PGSTAT_MAX_INTERVAL. */
+
+ if (pending_since > 0 &&
+ TimestampDifferenceExceeds(pending_since, now, PGSTAT_MAX_INTERVAL))
+ force = true;
+ }
+
+ /* for backends, update connection statistics */
if (MyBackendType == B_BACKEND)
- pgstat_send_connstats(disconnect, last_report);
+ pgstat_update_connstats(false);
- last_report = now;
+ /* don't wait for lock acquisition when !force */
+ nowait = !force;
- /*
- * Destroy pgStatTabHash before we start invalidating PgStat_TableEntry
- * entries it points to. (Should we fail partway through the loop below,
- * it's okay to have removed the hashtable already --- the only
- * consequence is we'd get multiple entries for the same table in the
- * pgStatTabList, and that's safe.)
- */
- if (pgStatTabHash)
- hash_destroy(pgStatTabHash);
- pgStatTabHash = NULL;
-
- /*
- * Scan through the TabStatusArray struct(s) to find tables that actually
- * have counts, and build messages to send. We have to separate shared
- * relations from regular ones because the databaseid field in the message
- * header has to depend on that.
- */
- regular_msg.m_databaseid = MyDatabaseId;
- shared_msg.m_databaseid = InvalidOid;
- regular_msg.m_nentries = 0;
- shared_msg.m_nentries = 0;
-
- for (tsa = pgStatTabList; tsa != NULL; tsa = tsa->tsa_next)
+ if (pgStatLocalHash)
{
- for (i = 0; i < tsa->tsa_used; i++)
+ int remains = 0;
+ pgstat_localhash_iterator i;
+ List *dbentlist = NIL;
+ ListCell *lc;
+ PgStatLocalHashEntry *lent;
+
+ /* Step 1: flush out other than database stats */
+ pgstat_localhash_start_iterate(pgStatLocalHash, &i);
+ while ((lent = pgstat_localhash_iterate(pgStatLocalHash, &i)) != NULL)
{
- PgStat_TableStatus *entry = &tsa->tsa_entries[i];
- PgStat_MsgTabstat *this_msg;
- PgStat_TableEntry *this_ent;
+ bool remove = false;
- /* Shouldn't have any pending transaction-dependent counts */
- Assert(entry->trans == NULL);
+ switch (lent->key.type)
+ {
+ case PGSTAT_TYPE_DB:
- /*
- * Ignore entries that didn't accumulate any actual counts, such
- * as indexes that were opened by the planner but not used.
- */
- if (memcmp(&entry->t_counts, &all_zeroes,
- sizeof(PgStat_TableCounts)) == 0)
+ /*
+ * flush_tabstat applies some of stats numbers of flushed
+ * entries into local database stats. Just remember the
+ * database entries for now then flush-out them later.
+ */
+ dbentlist = lappend(dbentlist, lent);
+ break;
+ case PGSTAT_TYPE_TABLE:
+ if (flush_tabstat(lent, nowait))
+ remove = true;
+ break;
+ case PGSTAT_TYPE_FUNCTION:
+ if (flush_funcstat(lent, nowait))
+ remove = true;
+ break;
+ case PGSTAT_TYPE_REPLSLOT:
+ /* We don't have that kind of local entry */
+ Assert(false);
+ }
+
+ if (!remove)
+ {
+ remains++;
continue;
+ }
- /*
- * OK, insert data into the appropriate message, and send if full.
- */
- this_msg = entry->t_shared ? &shared_msg : ®ular_msg;
- this_ent = &this_msg->m_entry[this_msg->m_nentries];
- this_ent->t_id = entry->t_id;
- memcpy(&this_ent->t_counts, &entry->t_counts,
- sizeof(PgStat_TableCounts));
- if (++this_msg->m_nentries >= PGSTAT_NUM_TABENTRIES)
+ /* Remove the successfully flushed entry */
+ pfree(lent->body);
+ lent->body = NULL;
+ pgstat_localhash_delete(pgStatLocalHash, lent->key);
+ }
+
+ /* Step 2: flush out database stats */
+ foreach(lc, dbentlist)
+ {
+ PgStatLocalHashEntry *lent = (PgStatLocalHashEntry *) lfirst(lc);
+
+ if (flush_dbstat(lent, nowait))
{
- pgstat_send_tabstat(this_msg);
- this_msg->m_nentries = 0;
+ remains--;
+ /* Remove the successfully flushed entry */
+ pfree(lent->body);
+ lent->body = NULL;
+ pgstat_localhash_delete(pgStatLocalHash, lent->key);
}
}
- /* zero out PgStat_TableStatus structs after use */
- MemSet(tsa->tsa_entries, 0,
- tsa->tsa_used * sizeof(PgStat_TableStatus));
- tsa->tsa_used = 0;
+ list_free(dbentlist);
+ dbentlist = NULL;
+
+ if (remains <= 0)
+ {
+ pgstat_localhash_destroy(pgStatLocalHash);
+ pgStatLocalHash = NULL;
+ }
}
+ /* flush wal stats */
+ flush_walstat(nowait);
+
+ /* flush SLRU stats */
+ flush_slrustat(nowait);
+
/*
- * Send partial messages. Make sure that any pending xact commit/abort
- * gets counted, even if there are no table stats to send.
+ * Publish the time of the last flush, but we don't notify the change of
+ * the timestamp itself. Readers will get sufficiently recent timestamp.
+ * If we failed to update the value, concurrent processes should have
+ * updated it to sufficiently recent time.
+ *
+ * XXX: The loop might be unnecessary for the reason above.
*/
- if (regular_msg.m_nentries > 0 ||
- pgStatXactCommit > 0 || pgStatXactRollback > 0)
- pgstat_send_tabstat(®ular_msg);
- if (shared_msg.m_nentries > 0)
- pgstat_send_tabstat(&shared_msg);
+ oldval = pg_atomic_read_u64(&StatsShmem->stats_timestamp);
- /* Now, send function statistics */
- pgstat_send_funcstats();
+ for (i = 0; i < 10; i++)
+ {
+ if (oldval >= now ||
+ pg_atomic_compare_exchange_u64(&StatsShmem->stats_timestamp,
+ &oldval, (uint64) now))
+ break;
+ }
- /* Send WAL statistics */
- pgstat_report_wal();
+ /*
+ * Some of the local stats may have not been flushed due to lock
+ * contention. If we have such pending local stats here, let the caller
+ * know the retry interval.
+ */
+ if (pgStatLocalHash != NULL || have_slrustats || walstats_pending())
+ {
+ /* Retain the epoch time */
+ if (pending_since == 0)
+ pending_since = now;
- /* Finally send SLRU statistics */
- pgstat_send_slru();
+ /* The interval is doubled at every retry. */
+ if (retry_interval == 0)
+ retry_interval = PGSTAT_RETRY_MIN_INTERVAL * 1000;
+ else
+ retry_interval = retry_interval * 2;
+
+ /*
+ * Determine the next retry interval so as not to get shorter than the
+ * previous interval.
+ */
+ if (!TimestampDifferenceExceeds(pending_since,
+ now + 2 * retry_interval,
+ PGSTAT_MAX_INTERVAL))
+ next_flush = now + retry_interval;
+ else
+ {
+ next_flush = pending_since + PGSTAT_MAX_INTERVAL * 1000;
+ retry_interval = next_flush - now;
+ }
+
+ return retry_interval / 1000;
+ }
+
+ /* Set the next time to update stats */
+ next_flush = now + PGSTAT_MIN_INTERVAL * 1000;
+ retry_interval = 0;
+ pending_since = 0;
+
+ return 0;
}
/*
- * Subroutine for pgstat_report_stat: finish and send a tabstat message
+ * flush_tabstat - flush out a local table stats entry
+ *
+ * Some of the stats numbers are copied to local database stats entry after
+ * successful flush-out.
+ *
+ * If nowait is true, this function returns false on lock failure. Otherwise
+ * this function always returns true.
+ *
+ * Returns true if the entry is successfully flushed out.
*/
-static void
-pgstat_send_tabstat(PgStat_MsgTabstat *tsmsg)
+static bool
+flush_tabstat(PgStatLocalHashEntry *ent, bool nowait)
{
- int n;
- int len;
+ static const PgStat_TableCounts all_zeroes;
+ Oid dboid; /* database OID of the table */
+ PgStat_TableStatus *lstats; /* local stats entry */
+ PgStat_StatTabEntry *shtabstats; /* table entry of shared stats */
+ PgStat_StatDBEntry *ldbstats; /* local database entry */
+
+ Assert(ent->key.type == PGSTAT_TYPE_TABLE);
+ lstats = (PgStat_TableStatus *) ent->body;
+ dboid = ent->key.databaseid;
+
+ /*
+ * Ignore entries that didn't accumulate any actual counts, such as
+ * indexes that were opened by the planner but not used.
+ */
+ if (memcmp(&lstats->t_counts, &all_zeroes,
+ sizeof(PgStat_TableCounts)) == 0)
+ {
+ /* This local entry is going to be dropped, delink from relcache. */
+ pgstat_delinkstats(lstats->relation);
+ return true;
+ }
+
+ /* find shared table stats entry corresponding to the local entry */
+ shtabstats = (PgStat_StatTabEntry *)
+ fetch_lock_statentry(PGSTAT_TYPE_TABLE, dboid, ent->key.objectid,
+ nowait);
+
+ if (shtabstats == NULL)
+ return false; /* failed to acquire lock, skip */
+
+ /* add the values to the shared entry. */
+ shtabstats->numscans += lstats->t_counts.t_numscans;
+ shtabstats->tuples_returned += lstats->t_counts.t_tuples_returned;
+ shtabstats->tuples_fetched += lstats->t_counts.t_tuples_fetched;
+ shtabstats->tuples_inserted += lstats->t_counts.t_tuples_inserted;
+ shtabstats->tuples_updated += lstats->t_counts.t_tuples_updated;
+ shtabstats->tuples_deleted += lstats->t_counts.t_tuples_deleted;
+ shtabstats->tuples_hot_updated += lstats->t_counts.t_tuples_hot_updated;
+
+ /*
+ * If table was truncated or vacuum/analyze has ran, first reset the
+ * live/dead counters.
+ */
+ if (lstats->t_counts.t_truncated)
+ {
+ shtabstats->n_live_tuples = 0;
+ shtabstats->n_dead_tuples = 0;
+ }
+
+ shtabstats->n_live_tuples += lstats->t_counts.t_delta_live_tuples;
+ shtabstats->n_dead_tuples += lstats->t_counts.t_delta_dead_tuples;
+ shtabstats->changes_since_analyze += lstats->t_counts.t_changed_tuples;
+ shtabstats->inserts_since_vacuum += lstats->t_counts.t_tuples_inserted;
+ shtabstats->blocks_fetched += lstats->t_counts.t_blocks_fetched;
+ shtabstats->blocks_hit += lstats->t_counts.t_blocks_hit;
+
+ /* Clamp n_live_tuples in case of negative delta_live_tuples */
+ shtabstats->n_live_tuples = Max(shtabstats->n_live_tuples, 0);
+ /* Likewise for n_dead_tuples */
+ shtabstats->n_dead_tuples = Max(shtabstats->n_dead_tuples, 0);
+
+ LWLockRelease(&shtabstats->header.lock);
+
+ /* The entry is successfully flushed so the same to add to database stats */
+ ldbstats = get_local_dbstat_entry(dboid);
+ ldbstats->counts.n_tuples_returned += lstats->t_counts.t_tuples_returned;
+ ldbstats->counts.n_tuples_fetched += lstats->t_counts.t_tuples_fetched;
+ ldbstats->counts.n_tuples_inserted += lstats->t_counts.t_tuples_inserted;
+ ldbstats->counts.n_tuples_updated += lstats->t_counts.t_tuples_updated;
+ ldbstats->counts.n_tuples_deleted += lstats->t_counts.t_tuples_deleted;
+ ldbstats->counts.n_blocks_fetched += lstats->t_counts.t_blocks_fetched;
+ ldbstats->counts.n_blocks_hit += lstats->t_counts.t_blocks_hit;
+
+ /* This local entry is going to be dropped, delink from relcache. */
+ pgstat_delinkstats(lstats->relation);
+
+ return true;
+}
+
+/*
+ * flush_funcstat - flush out a local function stats entry
+ *
+ * If nowait is true, this function returns false on lock failure. Otherwise
+ * this function always returns true.
+ *
+ * Returns true if the entry is successfully flushed out.
+ */
+static bool
+flush_funcstat(PgStatLocalHashEntry *ent, bool nowait)
+{
+ PgStat_BackendFunctionEntry *localent; /* local stats entry */
+ PgStat_StatFuncEntry *shfuncent = NULL; /* shared stats entry */
+
+ Assert(ent->key.type == PGSTAT_TYPE_FUNCTION);
+ localent = (PgStat_BackendFunctionEntry *) ent->body;
+
+ /* localent always has non-zero content */
+
+ /* find shared table stats entry corresponding to the local entry */
+ shfuncent = (PgStat_StatFuncEntry *)
+ fetch_lock_statentry(PGSTAT_TYPE_FUNCTION, MyDatabaseId,
+ ent->key.objectid, nowait);
+ if (shfuncent == NULL)
+ return false; /* failed to acquire lock, skip */
+
+ shfuncent->f_numcalls += localent->f_counts.f_numcalls;
+ shfuncent->f_total_time +=
+ INSTR_TIME_GET_MICROSEC(localent->f_counts.f_total_time);
+ shfuncent->f_self_time +=
+ INSTR_TIME_GET_MICROSEC(localent->f_counts.f_self_time);
+
+ LWLockRelease(&shfuncent->header.lock);
+
+ return true;
+}
+
+
+/*
+ * flush_dbstat - flush out a local database stats entry
+ *
+ * If nowait is true, this function returns false on lock failure. Otherwise
+ * this function always returns true.
+ *
+ * Returns true if the entry is successfully flushed out.
+ */
+#define PGSTAT_ACCUM_DBCOUNT(sh, lo, item) \
+ (sh)->counts.item += (lo)->counts.item
+
+static bool
+flush_dbstat(PgStatLocalHashEntry *ent, bool nowait)
+{
+ PgStat_StatDBEntry *localent;
+ PgStat_StatDBEntry *sharedent;
+
+ Assert(ent->key.type == PGSTAT_TYPE_DB);
+
+ localent = (PgStat_StatDBEntry *) &ent->body;
+
+ /* find shared database stats entry corresponding to the local entry */
+ sharedent = (PgStat_StatDBEntry *)
+ fetch_lock_statentry(PGSTAT_TYPE_DB, ent->key.databaseid, InvalidOid,
+ nowait);
+
+ if (!sharedent)
+ return false; /* failed to acquire lock, skip */
+
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_tuples_returned);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_tuples_fetched);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_tuples_inserted);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_tuples_updated);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_tuples_deleted);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_blocks_fetched);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_blocks_hit);
+
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_deadlocks);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_temp_bytes);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_temp_files);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_checksum_failures);
- /* It's unlikely we'd get here with no socket, but maybe not impossible */
- if (pgStatSock == PGINVALID_SOCKET)
- return;
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_sessions);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, total_session_time);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, total_active_time);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, total_idle_in_xact_time);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_sessions_abandoned);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_sessions_fatal);
+ PGSTAT_ACCUM_DBCOUNT(sharedent, localent, n_sessions_killed);
/*
- * Report and reset accumulated xact commit/rollback and I/O timings
- * whenever we send a normal tabstat message
+ * Accumulate xact commit/rollback and I/O timings to stats entry of the
+ * current database.
*/
- if (OidIsValid(tsmsg->m_databaseid))
+ if (OidIsValid(ent->key.databaseid))
{
- tsmsg->m_xact_commit = pgStatXactCommit;
- tsmsg->m_xact_rollback = pgStatXactRollback;
- tsmsg->m_block_read_time = pgStatBlockReadTime;
- tsmsg->m_block_write_time = pgStatBlockWriteTime;
+ sharedent->counts.n_xact_commit += pgStatXactCommit;
+ sharedent->counts.n_xact_rollback += pgStatXactRollback;
+ sharedent->counts.n_block_read_time += pgStatBlockReadTime;
+ sharedent->counts.n_block_write_time += pgStatBlockWriteTime;
pgStatXactCommit = 0;
pgStatXactRollback = 0;
pgStatBlockReadTime = 0;
@@ -1011,281 +1565,139 @@ pgstat_send_tabstat(PgStat_MsgTabstat *tsmsg)
}
else
{
- tsmsg->m_xact_commit = 0;
- tsmsg->m_xact_rollback = 0;
- tsmsg->m_block_read_time = 0;
- tsmsg->m_block_write_time = 0;
+ sharedent->counts.n_xact_commit = 0;
+ sharedent->counts.n_xact_rollback = 0;
+ sharedent->counts.n_block_read_time = 0;
+ sharedent->counts.n_block_write_time = 0;
}
- n = tsmsg->m_nentries;
- len = offsetof(PgStat_MsgTabstat, m_entry[0]) +
- n * sizeof(PgStat_TableEntry);
+ LWLockRelease(&sharedent->header.lock);
- pgstat_setheader(&tsmsg->m_hdr, PGSTAT_MTYPE_TABSTAT);
- pgstat_send(tsmsg, len);
+ return true;
}
-/*
- * Subroutine for pgstat_report_stat: populate and send a function stat message
- */
-static void
-pgstat_send_funcstats(void)
-{
- /* we assume this inits to all zeroes: */
- static const PgStat_FunctionCounts all_zeroes;
-
- PgStat_MsgFuncstat msg;
- PgStat_BackendFunctionEntry *entry;
- HASH_SEQ_STATUS fstat;
-
- if (pgStatFunctions == NULL)
- return;
-
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_FUNCSTAT);
- msg.m_databaseid = MyDatabaseId;
- msg.m_nentries = 0;
-
- hash_seq_init(&fstat, pgStatFunctions);
- while ((entry = (PgStat_BackendFunctionEntry *) hash_seq_search(&fstat)) != NULL)
- {
- PgStat_FunctionEntry *m_ent;
-
- /* Skip it if no counts accumulated since last time */
- if (memcmp(&entry->f_counts, &all_zeroes,
- sizeof(PgStat_FunctionCounts)) == 0)
- continue;
-
- /* need to convert format of time accumulators */
- m_ent = &msg.m_entry[msg.m_nentries];
- m_ent->f_id = entry->f_id;
- m_ent->f_numcalls = entry->f_counts.f_numcalls;
- m_ent->f_total_time = INSTR_TIME_GET_MICROSEC(entry->f_counts.f_total_time);
- m_ent->f_self_time = INSTR_TIME_GET_MICROSEC(entry->f_counts.f_self_time);
-
- if (++msg.m_nentries >= PGSTAT_NUM_FUNCENTRIES)
- {
- pgstat_send(&msg, offsetof(PgStat_MsgFuncstat, m_entry[0]) +
- msg.m_nentries * sizeof(PgStat_FunctionEntry));
- msg.m_nentries = 0;
- }
-
- /* reset the entry's counts */
- MemSet(&entry->f_counts, 0, sizeof(PgStat_FunctionCounts));
- }
-
- if (msg.m_nentries > 0)
- pgstat_send(&msg, offsetof(PgStat_MsgFuncstat, m_entry[0]) +
- msg.m_nentries * sizeof(PgStat_FunctionEntry));
-
- have_function_stats = false;
-}
-
-
/* ----------
* pgstat_vacuum_stat() -
*
- * Will tell the collector about objects he can get rid of.
+ * Delete shared stat entries that are not in system catalogs.
+ *
+ * To avoid holding exclusive lock on dshash for a long time, the process is
+ * performed in three steps.
+ *
+ * 1: Collect existent oids of every kind of object.
+ * 2: Collect victim entries by scanning with shared lock.
+ * 3: Try removing every nominated entry without waiting for lock.
+ *
+ * As the consequence of the last step, some entries may be left alone due to
+ * lock failure, but as explained by the comment of pgstat_vacuum_stat, they
+ * will be deleted by later vacuums.
* ----------
*/
void
pgstat_vacuum_stat(void)
{
- HTAB *htab;
- PgStat_MsgTabpurge msg;
- PgStat_MsgFuncpurge f_msg;
- HASH_SEQ_STATUS hstat;
- PgStat_StatDBEntry *dbentry;
- PgStat_StatTabEntry *tabentry;
- PgStat_StatFuncEntry *funcentry;
- int len;
-
- if (pgStatSock == PGINVALID_SOCKET)
- return;
-
- /*
- * If not done for this transaction, read the statistics collector stats
- * file into some hash tables.
- */
- backend_read_statsfile();
-
- /*
- * Read pg_database and make a list of OIDs of all existing databases
- */
- htab = pgstat_collect_oids(DatabaseRelationId, Anum_pg_database_oid);
-
- /*
- * Search the database hash table for dead databases and tell the
- * collector to drop them.
- */
- hash_seq_init(&hstat, pgStatDBHash);
- while ((dbentry = (PgStat_StatDBEntry *) hash_seq_search(&hstat)) != NULL)
- {
- Oid dbid = dbentry->databaseid;
-
- CHECK_FOR_INTERRUPTS();
-
- /* the DB entry for shared tables (with InvalidOid) is never dropped */
- if (OidIsValid(dbid) &&
- hash_search(htab, (void *) &dbid, HASH_FIND, NULL) == NULL)
- pgstat_drop_database(dbid);
- }
-
- /* Clean up */
- hash_destroy(htab);
-
- /*
- * Lookup our own database entry; if not found, nothing more to do.
- */
- dbentry = (PgStat_StatDBEntry *) hash_search(pgStatDBHash,
- (void *) &MyDatabaseId,
- HASH_FIND, NULL);
- if (dbentry == NULL || dbentry->tables == NULL)
+ pgstat_oid_hash *dbids; /* database ids */
+ pgstat_oid_hash *relids; /* relation ids in the current database */
+ pgstat_oid_hash *funcids; /* function ids in the current database */
+ int nvictims = 0; /* # of entries of the above */
+ dshash_seq_status dshstat;
+ PgStatHashEntry *ent;
+
+ /* we don't collect stats under standalone mode */
+ if (!IsUnderPostmaster)
return;
- /*
- * Similarly to above, make a list of all known relations in this DB.
- */
- htab = pgstat_collect_oids(RelationRelationId, Anum_pg_class_oid);
+ /* collect oids of existent objects */
+ dbids = collect_oids(DatabaseRelationId, Anum_pg_database_oid);
+ relids = collect_oids(RelationRelationId, Anum_pg_class_oid);
+ funcids = collect_oids(ProcedureRelationId, Anum_pg_proc_oid);
- /*
- * Initialize our messages table counter to zero
- */
- msg.m_nentries = 0;
+ nvictims = 0;
- /*
- * Check for all tables listed in stats hashtable if they still exist.
- */
- hash_seq_init(&hstat, dbentry->tables);
- while ((tabentry = (PgStat_StatTabEntry *) hash_seq_search(&hstat)) != NULL)
+ /* some of the dshash entries are to be removed, take exclusive lock. */
+ dshash_seq_init(&dshstat, pgStatSharedHash, true);
+ while ((ent = dshash_seq_next(&dshstat)) != NULL)
{
- Oid tabid = tabentry->tableid;
+ pgstat_oid_hash *oidhash;
+ Oid key;
CHECK_FOR_INTERRUPTS();
- if (hash_search(htab, (void *) &tabid, HASH_FIND, NULL) != NULL)
+ /*
+ * Don't drop entries for other than database objects not of the
+ * current database.
+ */
+ if (ent->key.type != PGSTAT_TYPE_DB &&
+ ent->key.databaseid != MyDatabaseId)
continue;
- /*
- * Not there, so add this table's Oid to the message
- */
- msg.m_tableid[msg.m_nentries++] = tabid;
-
- /*
- * If the message is full, send it out and reinitialize to empty
- */
- if (msg.m_nentries >= PGSTAT_NUM_TABPURGE)
- {
- len = offsetof(PgStat_MsgTabpurge, m_tableid[0])
- + msg.m_nentries * sizeof(Oid);
-
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_TABPURGE);
- msg.m_databaseid = MyDatabaseId;
- pgstat_send(&msg, len);
-
- msg.m_nentries = 0;
- }
- }
-
- /*
- * Send the rest
- */
- if (msg.m_nentries > 0)
- {
- len = offsetof(PgStat_MsgTabpurge, m_tableid[0])
- + msg.m_nentries * sizeof(Oid);
-
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_TABPURGE);
- msg.m_databaseid = MyDatabaseId;
- pgstat_send(&msg, len);
- }
-
- /* Clean up */
- hash_destroy(htab);
-
- /*
- * Now repeat the above steps for functions. However, we needn't bother
- * in the common case where no function stats are being collected.
- */
- if (dbentry->functions != NULL &&
- hash_get_num_entries(dbentry->functions) > 0)
- {
- htab = pgstat_collect_oids(ProcedureRelationId, Anum_pg_proc_oid);
-
- pgstat_setheader(&f_msg.m_hdr, PGSTAT_MTYPE_FUNCPURGE);
- f_msg.m_databaseid = MyDatabaseId;
- f_msg.m_nentries = 0;
-
- hash_seq_init(&hstat, dbentry->functions);
- while ((funcentry = (PgStat_StatFuncEntry *) hash_seq_search(&hstat)) != NULL)
+ switch (ent->key.type)
{
- Oid funcid = funcentry->functionid;
-
- CHECK_FOR_INTERRUPTS();
-
- if (hash_search(htab, (void *) &funcid, HASH_FIND, NULL) != NULL)
+ case PGSTAT_TYPE_DB:
+ /* don't remove database entry for shared tables */
+ if (ent->key.databaseid == 0)
+ continue;
+ oidhash = dbids;
+ key = ent->key.databaseid;
+ break;
+
+ case PGSTAT_TYPE_TABLE:
+ oidhash = relids;
+ key = ent->key.objectid;
+ break;
+
+ case PGSTAT_TYPE_FUNCTION:
+ oidhash = funcids;
+ key = ent->key.objectid;
+ break;
+
+ case PGSTAT_TYPE_REPLSLOT:
+
+ /*
+ * We don't bother vacuumming this kind of entries because the
+ * number of entries is quite small and entries are likely to
+ * be reused soon.
+ */
continue;
-
- /*
- * Not there, so add this function's Oid to the message
- */
- f_msg.m_functionid[f_msg.m_nentries++] = funcid;
-
- /*
- * If the message is full, send it out and reinitialize to empty
- */
- if (f_msg.m_nentries >= PGSTAT_NUM_FUNCPURGE)
- {
- len = offsetof(PgStat_MsgFuncpurge, m_functionid[0])
- + f_msg.m_nentries * sizeof(Oid);
-
- pgstat_send(&f_msg, len);
-
- f_msg.m_nentries = 0;
- }
}
- /*
- * Send the rest
- */
- if (f_msg.m_nentries > 0)
- {
- len = offsetof(PgStat_MsgFuncpurge, m_functionid[0])
- + f_msg.m_nentries * sizeof(Oid);
-
- pgstat_send(&f_msg, len);
- }
+ /* Skip existent objects. */
+ if (pgstat_oid_lookup(oidhash, key) != NULL)
+ continue;
- hash_destroy(htab);
+ /* drop this etnry */
+ delete_current_stats_entry(&dshstat);
+ nvictims++;
}
+ dshash_seq_term(&dshstat);
+ pgstat_oid_destroy(dbids);
+ pgstat_oid_destroy(relids);
+ pgstat_oid_destroy(funcids);
+
+ if (nvictims > 0)
+ pg_atomic_add_fetch_u64(&StatsShmem->gc_count, 1);
}
-
/* ----------
- * pgstat_collect_oids() -
+ * collect_oids() -
*
* Collect the OIDs of all objects listed in the specified system catalog
- * into a temporary hash table. Caller should hash_destroy the result
+ * into a temporary hash table. Caller should pgsstat_oid_destroy the result
* when done with it. (However, we make the table in CurrentMemoryContext
* so that it will be freed properly in event of an error.)
* ----------
*/
-static HTAB *
-pgstat_collect_oids(Oid catalogid, AttrNumber anum_oid)
+static pgstat_oid_hash *
+collect_oids(Oid catalogid, AttrNumber anum_oid)
{
- HTAB *htab;
- HASHCTL hash_ctl;
+ pgstat_oid_hash *rethash;
Relation rel;
TableScanDesc scan;
HeapTuple tup;
Snapshot snapshot;
- hash_ctl.keysize = sizeof(Oid);
- hash_ctl.entrysize = sizeof(Oid);
- hash_ctl.hcxt = CurrentMemoryContext;
- htab = hash_create("Temporary table of OIDs",
- PGSTAT_TAB_HASH_SIZE,
- &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ rethash = pgstat_oid_create(CurrentMemoryContext,
+ PGSTAT_TABLE_HASH_SIZE, NULL);
rel = table_open(catalogid, AccessShareLock);
snapshot = RegisterSnapshot(GetLatestSnapshot());
@@ -1294,123 +1706,60 @@ pgstat_collect_oids(Oid catalogid, AttrNumber anum_oid)
{
Oid thisoid;
bool isnull;
+ bool found;
thisoid = heap_getattr(tup, anum_oid, RelationGetDescr(rel), &isnull);
Assert(!isnull);
CHECK_FOR_INTERRUPTS();
- (void) hash_search(htab, (void *) &thisoid, HASH_ENTER, NULL);
+ pgstat_oid_insert(rethash, thisoid, &found);
}
table_endscan(scan);
UnregisterSnapshot(snapshot);
table_close(rel, AccessShareLock);
- return htab;
+ return rethash;
}
-
/* ----------
* pgstat_drop_database() -
*
- * Tell the collector that we just dropped a database.
- * (If the message gets lost, we will still clean the dead DB eventually
- * via future invocations of pgstat_vacuum_stat().)
- * ----------
+ * Remove entry for the database that we just dropped.
+ *
+ * Some entries might be left alone due to lock failure or some stats are
+ * flushed after this but we will still clean the dead DB eventually via
+ * future invocations of pgstat_vacuum_stat().
+ * ----------
*/
void
pgstat_drop_database(Oid databaseid)
{
- PgStat_MsgDropdb msg;
+ dshash_seq_status hstat;
+ PgStatHashEntry *p;
- if (pgStatSock == PGINVALID_SOCKET)
- return;
-
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_DROPDB);
- msg.m_databaseid = databaseid;
- pgstat_send(&msg, sizeof(msg));
-}
-
-
-/* ----------
- * pgstat_drop_relation() -
- *
- * Tell the collector that we just dropped a relation.
- * (If the message gets lost, we will still clean the dead entry eventually
- * via future invocations of pgstat_vacuum_stat().)
- *
- * Currently not used for lack of any good place to call it; we rely
- * entirely on pgstat_vacuum_stat() to clean out stats for dead rels.
- * ----------
- */
-#ifdef NOT_USED
-void
-pgstat_drop_relation(Oid relid)
-{
- PgStat_MsgTabpurge msg;
- int len;
-
- if (pgStatSock == PGINVALID_SOCKET)
- return;
-
- msg.m_tableid[0] = relid;
- msg.m_nentries = 1;
-
- len = offsetof(PgStat_MsgTabpurge, m_tableid[0]) + sizeof(Oid);
-
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_TABPURGE);
- msg.m_databaseid = MyDatabaseId;
- pgstat_send(&msg, len);
-}
-#endif /* NOT_USED */
-
-
-/* ----------
- * pgstat_send_connstats() -
- *
- * Tell the collector about session statistics.
- * The parameter "disconnect" will be true when the backend exits.
- * "last_report" is the last time we were called (0 if never).
- * ----------
- */
-static void
-pgstat_send_connstats(bool disconnect, TimestampTz last_report)
-{
- PgStat_MsgConn msg;
- long secs;
- int usecs;
+ Assert(OidIsValid(databaseid));
- if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ if (!IsUnderPostmaster || !pgStatSharedHash)
return;
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_CONNECTION);
- msg.m_databaseid = MyDatabaseId;
-
- /* session time since the last report */
- TimestampDifference(((last_report == 0) ? MyStartTimestamp : last_report),
- GetCurrentTimestamp(),
- &secs, &usecs);
- msg.m_session_time = secs * 1000000 + usecs;
-
- msg.m_disconnect = disconnect ? pgStatSessionEndCause : DISCONNECT_NOT_YET;
-
- msg.m_active_time = pgStatActiveTime;
- pgStatActiveTime = 0;
-
- msg.m_idle_in_xact_time = pgStatTransactionIdleTime;
- pgStatTransactionIdleTime = 0;
-
- /* report a new session only the first time */
- msg.m_count = (last_report == 0) ? 1 : 0;
-
- pgstat_send(&msg, sizeof(PgStat_MsgConn));
+ /* some of the dshash entries are to be removed, take exclusive lock. */
+ dshash_seq_init(&hstat, pgStatSharedHash, true);
+ while ((p = dshash_seq_next(&hstat)) != NULL)
+ {
+ if (p->key.databaseid == MyDatabaseId)
+ delete_current_stats_entry(&hstat);
+ }
+ dshash_seq_term(&hstat);
+
+ /* Let readers run a garbage collection of local hashes */
+ pg_atomic_add_fetch_u64(&StatsShmem->gc_count, 1);
}
-
/* ----------
* pgstat_reset_counters() -
*
- * Tell the statistics collector to reset counters for our database.
+ * Reset counters for our database.
*
* Permission checking for this function is managed through the normal
* GRANT system.
@@ -1419,53 +1768,148 @@ pgstat_send_connstats(bool disconnect, TimestampTz last_report)
void
pgstat_reset_counters(void)
{
- PgStat_MsgResetcounter msg;
+ dshash_seq_status hstat;
+ PgStatHashEntry *p;
- if (pgStatSock == PGINVALID_SOCKET)
+ if (!IsUnderPostmaster || !pgStatSharedHash)
return;
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETCOUNTER);
- msg.m_databaseid = MyDatabaseId;
- pgstat_send(&msg, sizeof(msg));
+ /* dshash entry is not modified, take shared lock */
+ dshash_seq_init(&hstat, pgStatSharedHash, false);
+ while ((p = dshash_seq_next(&hstat)) != NULL)
+ {
+ PgStat_StatEntryHeader *header;
+
+ if (p->key.databaseid != MyDatabaseId)
+ continue;
+
+ header = dsa_get_address(area, p->body);
+
+ LWLockAcquire(&header->lock, LW_EXCLUSIVE);
+ memset(PGSTAT_SHENT_BODY(header), 0,
+ PGSTAT_SHENT_BODY_LEN(p->key.type));
+
+ if (p->key.type == PGSTAT_TYPE_DB)
+ {
+ PgStat_StatDBEntry *dbstat = (PgStat_StatDBEntry *) header;
+
+ dbstat->stat_reset_timestamp = GetCurrentTimestamp();
+ }
+ LWLockRelease(&header->lock);
+ }
+ dshash_seq_term(&hstat);
+
+ /* Invalidate the simple cache keys */
+ cached_dbent_key = stathashkey_zero;
+ cached_tabent_key = stathashkey_zero;
+ cached_funcent_key = stathashkey_zero;
+}
+
+/*
+ * pgstat_copy_global_stats - helper function for functions
+ * pgstat_fetch_stat_*() and pgstat_reset_shared_counters().
+ *
+ * Copies out the specified memory area following change-count protocol.
+ */
+static inline void
+pgstat_copy_global_stats(void *dst, void *src, size_t len,
+ pg_atomic_uint32 *count)
+{
+ int before_changecount;
+ int after_changecount;
+
+ after_changecount = pg_atomic_read_u32(count);
+
+ do
+ {
+ before_changecount = after_changecount;
+ memcpy(dst, src, len);
+ after_changecount = pg_atomic_read_u32(count);
+ } while ((before_changecount & 1) == 1 ||
+ after_changecount != before_changecount);
}
/* ----------
* pgstat_reset_shared_counters() -
*
- * Tell the statistics collector to reset cluster-wide shared counters.
+ * Reset cluster-wide shared counters.
*
* Permission checking for this function is managed through the normal
* GRANT system.
+ *
+ * We don't scribble on shared stats while resetting to avoid locking on
+ * shared stats struct. Instead, just record the current counters in another
+ * shared struct, which is protected by StatsLock. See
+ * pgstat_fetch_stat_(archiver|bgwriter|checkpointer) for the reader side.
* ----------
*/
void
pgstat_reset_shared_counters(const char *target)
{
- PgStat_MsgResetsharedcounter msg;
-
- if (pgStatSock == PGINVALID_SOCKET)
- return;
+ TimestampTz now = GetCurrentTimestamp();
+ PgStat_Shared_Reset_Target t;
if (strcmp(target, "archiver") == 0)
- msg.m_resettarget = RESET_ARCHIVER;
+ t = RESET_ARCHIVER;
else if (strcmp(target, "bgwriter") == 0)
- msg.m_resettarget = RESET_BGWRITER;
+ t = RESET_BGWRITER;
else if (strcmp(target, "wal") == 0)
- msg.m_resettarget = RESET_WAL;
+ t = RESET_WAL;
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("unrecognized reset target: \"%s\"", target),
errhint("Target must be \"archiver\", \"bgwriter\" or \"wal\".")));
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSHAREDCOUNTER);
- pgstat_send(&msg, sizeof(msg));
+ /* Reset statistics for the cluster. */
+ LWLockAcquire(StatsLock, LW_EXCLUSIVE);
+
+ switch (t)
+ {
+ case RESET_ARCHIVER:
+ pgstat_copy_global_stats(&StatsShmem->archiver_reset_offset,
+ &StatsShmem->archiver_stats,
+ sizeof(PgStat_Archiver),
+ &StatsShmem->archiver_changecount);
+ StatsShmem->archiver_reset_offset.stat_reset_timestamp = now;
+ cached_archiverstats_is_valid = false;
+ break;
+
+ case RESET_BGWRITER:
+ pgstat_copy_global_stats(&StatsShmem->bgwriter_reset_offset,
+ &StatsShmem->bgwriter_stats,
+ sizeof(PgStat_BgWriter),
+ &StatsShmem->bgwriter_changecount);
+ pgstat_copy_global_stats(&StatsShmem->checkpointer_reset_offset,
+ &StatsShmem->checkpointer_stats,
+ sizeof(PgStat_CheckPointer),
+ &StatsShmem->checkpointer_changecount);
+ StatsShmem->bgwriter_reset_offset.stat_reset_timestamp = now;
+ cached_bgwriterstats_is_valid = false;
+ cached_checkpointerstats_is_valid = false;
+ break;
+
+ case RESET_WAL:
+
+ /*
+ * Differntly from the two above, WAL statistics has many writer
+ * processes and protected by wal_stats_lock.
+ */
+ LWLockAcquire(&StatsShmem->wal_stats_lock, LW_EXCLUSIVE);
+ MemSet(&StatsShmem->wal_stats, 0, sizeof(PgStat_Wal));
+ StatsShmem->wal_stats.stat_reset_timestamp = now;
+ LWLockRelease(&StatsShmem->wal_stats_lock);
+ cached_walstats_is_valid = false;
+ break;
+ }
+
+ LWLockRelease(StatsLock);
}
/* ----------
* pgstat_reset_single_counter() -
*
- * Tell the statistics collector to reset a single counter.
+ * Reset a single counter.
*
* Permission checking for this function is managed through the normal
* GRANT system.
@@ -1474,17 +1918,37 @@ pgstat_reset_shared_counters(const char *target)
void
pgstat_reset_single_counter(Oid objoid, PgStat_Single_Reset_Type type)
{
- PgStat_MsgResetsinglecounter msg;
+ PgStat_StatEntryHeader *header;
+ PgStat_StatDBEntry *dbentry;
+ PgStatTypes stattype;
+ TimestampTz ts;
- if (pgStatSock == PGINVALID_SOCKET)
- return;
+ dbentry = (PgStat_StatDBEntry *)
+ get_stat_entry(PGSTAT_TYPE_DB, MyDatabaseId, InvalidOid, false, false,
+ NULL);
+ Assert(dbentry);
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSINGLECOUNTER);
- msg.m_databaseid = MyDatabaseId;
- msg.m_resettype = type;
- msg.m_objectid = objoid;
+ /* Set the reset timestamp for the whole database */
+ ts = GetCurrentTimestamp();
+ LWLockAcquire(&dbentry->header.lock, LW_EXCLUSIVE);
+ dbentry->stat_reset_timestamp = ts;
+ LWLockRelease(&dbentry->header.lock);
- pgstat_send(&msg, sizeof(msg));
+ /* Remove object if it exists, ignore if not */
+ switch (type)
+ {
+ case RESET_TABLE:
+ stattype = PGSTAT_TYPE_TABLE;
+ break;
+ case RESET_FUNCTION:
+ stattype = PGSTAT_TYPE_FUNCTION;
+ }
+
+ header = get_stat_entry(stattype, MyDatabaseId, objoid, false, false, NULL);
+
+ LWLockAcquire(&header->lock, LW_EXCLUSIVE);
+ memset(PGSTAT_SHENT_BODY(header), 0, PGSTAT_SHENT_BODY_LEN(stattype));
+ LWLockRelease(&header->lock);
}
/* ----------
@@ -1500,15 +1964,42 @@ pgstat_reset_single_counter(Oid objoid, PgStat_Single_Reset_Type type)
void
pgstat_reset_slru_counter(const char *name)
{
- PgStat_MsgResetslrucounter msg;
+ int i;
+ TimestampTz ts = GetCurrentTimestamp();
+ uint32 assert_changecount;
- if (pgStatSock == PGINVALID_SOCKET)
- return;
+ PG_USED_FOR_ASSERTS_ONLY;
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSLRUCOUNTER);
- msg.m_index = (name) ? pgstat_slru_index(name) : -1;
+ if (name)
+ {
+ i = pgstat_slru_index(name);
+ LWLockAcquire(&StatsShmem->slru_stats.lock, LW_EXCLUSIVE);
+ assert_changecount =
+ pg_atomic_fetch_add_u32(&StatsShmem->slru_changecount, 1);
+ Assert((assert_changecount & 1) == 0);
+ MemSet(&StatsShmem->slru_stats.entry[i], 0,
+ sizeof(PgStat_SLRUStats));
+ StatsShmem->slru_stats.entry[i].stat_reset_timestamp = ts;
+ pg_atomic_add_fetch_u32(&StatsShmem->slru_changecount, 1);
+ LWLockRelease(&StatsShmem->slru_stats.lock);
+ }
+ else
+ {
+ LWLockAcquire(&StatsShmem->slru_stats.lock, LW_EXCLUSIVE);
+ assert_changecount =
+ pg_atomic_fetch_add_u32(&StatsShmem->slru_changecount, 1);
+ Assert((assert_changecount & 1) == 0);
+ for (i = 0; i < SLRU_NUM_ELEMENTS; i++)
+ {
+ MemSet(&StatsShmem->slru_stats.entry[i], 0,
+ sizeof(PgStat_SLRUStats));
+ StatsShmem->slru_stats.entry[i].stat_reset_timestamp = ts;
+ }
+ pg_atomic_add_fetch_u32(&StatsShmem->slru_changecount, 1);
+ LWLockRelease(&StatsShmem->slru_stats.lock);
+ }
- pgstat_send(&msg, sizeof(msg));
+ cached_slrustats_is_valid = false;
}
/* ----------
@@ -1524,20 +2015,19 @@ pgstat_reset_slru_counter(const char *name)
void
pgstat_reset_replslot_counter(const char *name)
{
- PgStat_MsgResetreplslotcounter msg;
+ int startidx;
+ int endidx;
+ int i;
+ TimestampTz ts;
- if (pgStatSock == PGINVALID_SOCKET)
+ if (!IsUnderPostmaster || !pgStatSharedHash)
return;
if (name)
{
ReplicationSlot *slot;
- /*
- * Check if the slot exits with the given name. It is possible that by
- * the time this message is executed the slot is dropped but at least
- * this check will ensure that the given name is for a valid slot.
- */
+ /* Check if the slot exits with the given name. */
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
slot = SearchNamedReplicationSlot(name);
LWLockRelease(ReplicationSlotControlLock);
@@ -1555,15 +2045,36 @@ pgstat_reset_replslot_counter(const char *name)
if (SlotIsPhysical(slot))
return;
- strlcpy(msg.m_slotname, name, NAMEDATALEN);
- msg.clearall = false;
+ /* reset this one entry */
+ startidx = endidx = slot - ReplicationSlotCtl->replication_slots;
}
else
- msg.clearall = true;
+ {
+ /* reset all existent entries */
+ startidx = 0;
+ endidx = max_replication_slots - 1;
+ }
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETREPLSLOTCOUNTER);
+ ts = GetCurrentTimestamp();
+ for (i = startidx; i <= endidx; i++)
+ {
+ PgStat_ReplSlot *shent;
- pgstat_send(&msg, sizeof(msg));
+ shent = (PgStat_ReplSlot *)
+ get_stat_entry(PGSTAT_TYPE_REPLSLOT,
+ MyDatabaseId, i, false, false, NULL);
+
+ /* Skip non-existent entries */
+ if (!shent)
+ continue;
+
+ LWLockAcquire(&shent->header.lock, LW_EXCLUSIVE);
+ memset(&shent->spill_txns, 0,
+ offsetof(PgStat_ReplSlot, stat_reset_timestamp) -
+ offsetof(PgStat_ReplSlot, spill_txns));
+ shent->stat_reset_timestamp = ts;
+ LWLockRelease(&shent->header.lock);
+ }
}
/* ----------
@@ -1577,48 +2088,94 @@ pgstat_reset_replslot_counter(const char *name)
void
pgstat_report_autovac(Oid dboid)
{
- PgStat_MsgAutovacStart msg;
+ PgStat_StatDBEntry *dbentry;
+ TimestampTz ts;
- if (pgStatSock == PGINVALID_SOCKET)
+ /* return if activity stats is not active */
+ if (!area)
return;
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_AUTOVAC_START);
- msg.m_databaseid = dboid;
- msg.m_start_time = GetCurrentTimestamp();
+ /*
+ * End-of-vacuum is reported instantly. Report the start the same way for
+ * consistency. Vacuum doesn't run frequently and is a long-lasting
+ * operation so it doesn't matter if we get blocked here a little.
+ */
+ dbentry = (PgStat_StatDBEntry *)
+ get_stat_entry(PGSTAT_TYPE_DB, dboid, InvalidOid, false, true, NULL);
- pgstat_send(&msg, sizeof(msg));
+ ts = GetCurrentTimestamp();;
+ LWLockAcquire(&dbentry->header.lock, LW_EXCLUSIVE);
+ dbentry->last_autovac_time = ts;
+ LWLockRelease(&dbentry->header.lock);
}
/* ---------
* pgstat_report_vacuum() -
*
- * Tell the collector about the table we just vacuumed.
+ * Report about the table we just vacuumed.
* ---------
*/
void
pgstat_report_vacuum(Oid tableoid, bool shared,
PgStat_Counter livetuples, PgStat_Counter deadtuples)
{
- PgStat_MsgVacuum msg;
+ PgStat_StatTabEntry *tabentry;
+ Oid dboid = (shared ? InvalidOid : MyDatabaseId);
+ TimestampTz ts;
- if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ /* return if we are not collecting stats */
+ if (!area)
return;
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_VACUUM);
- msg.m_databaseid = shared ? InvalidOid : MyDatabaseId;
- msg.m_tableoid = tableoid;
- msg.m_autovacuum = IsAutoVacuumWorkerProcess();
- msg.m_vacuumtime = GetCurrentTimestamp();
- msg.m_live_tuples = livetuples;
- msg.m_dead_tuples = deadtuples;
- pgstat_send(&msg, sizeof(msg));
+ /* Store the data in the table's hash table entry. */
+ ts = GetCurrentTimestamp();
+
+ /*
+ * Differently from ordinary operations, maintenance commands take longer
+ * time and getting blocked at the end of work doesn't matter.
+ * Furthermore, this can prevent the stats updates made by the
+ * transactions that ends after this vacuum from being canceled by a
+ * delayed vacuum report. Update shared stats entry directly for the above
+ * reasons.
+ */
+ tabentry = (PgStat_StatTabEntry *)
+ get_stat_entry(PGSTAT_TYPE_TABLE, dboid, tableoid, false, true, NULL);
+
+ LWLockAcquire(&tabentry->header.lock, LW_EXCLUSIVE);
+ tabentry->n_live_tuples = livetuples;
+ tabentry->n_dead_tuples = deadtuples;
+
+ /*
+ * It is quite possible that a non-aggressive VACUUM ended up skipping
+ * various pages, however, we'll zero the insert counter here regardless.
+ * It's currently used only to track when we need to perform an "insert"
+ * autovacuum, which are mainly intended to freeze newly inserted tuples.
+ * Zeroing this may just mean we'll not try to vacuum the table again
+ * until enough tuples have been inserted to trigger another insert
+ * autovacuum. An anti-wraparound autovacuum will catch any persistent
+ * stragglers.
+ */
+ tabentry->inserts_since_vacuum = 0;
+
+ if (IsAutoVacuumWorkerProcess())
+ {
+ tabentry->autovac_vacuum_timestamp = ts;
+ tabentry->autovac_vacuum_count++;
+ }
+ else
+ {
+ tabentry->vacuum_timestamp = ts;
+ tabentry->vacuum_count++;
+ }
+
+ LWLockRelease(&tabentry->header.lock);
}
/* --------
* pgstat_report_analyze() -
*
- * Tell the collector about the table we just analyzed.
+ * Report about the table we just analyzed.
*
* Caller must provide new live- and dead-tuples estimates, as well as a
* flag indicating whether to reset the changes_since_analyze counter.
@@ -1629,9 +2186,11 @@ pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter)
{
- PgStat_MsgAnalyze msg;
+ PgStat_StatTabEntry *tabentry;
+ Oid dboid = (rel->rd_rel->relisshared ? InvalidOid : MyDatabaseId);
- if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ /* return if we are not collecting stats */
+ if (!area)
return;
/*
@@ -1639,10 +2198,10 @@ pgstat_report_analyze(Relation rel,
* already inserted and/or deleted rows in the target table. ANALYZE will
* have counted such rows as live or dead respectively. Because we will
* report our counts of such rows at transaction end, we should subtract
- * off these counts from what we send to the collector now, else they'll
- * be double-counted after commit. (This approach also ensures that the
- * collector ends up with the right numbers if we abort instead of
- * committing.)
+ * off these counts from what is already written to shared stats now, else
+ * they'll be double-counted after commit. (This approach also ensures
+ * that the shared stats ends up with the right numbers if we abort
+ * instead of committing.)
*/
if (rel->pgstat_info != NULL)
{
@@ -1660,137 +2219,224 @@ pgstat_report_analyze(Relation rel,
deadtuples = Max(deadtuples, 0);
}
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_ANALYZE);
- msg.m_databaseid = rel->rd_rel->relisshared ? InvalidOid : MyDatabaseId;
- msg.m_tableoid = RelationGetRelid(rel);
- msg.m_autovacuum = IsAutoVacuumWorkerProcess();
- msg.m_resetcounter = resetcounter;
- msg.m_analyzetime = GetCurrentTimestamp();
- msg.m_live_tuples = livetuples;
- msg.m_dead_tuples = deadtuples;
- pgstat_send(&msg, sizeof(msg));
+ /*
+ * Differently from ordinary operations, maintenance commands take longer
+ * time and getting blocked at the end of work doesn't matter.
+ * Furthermore, this can prevent the stats updates made by the
+ * transactions that ends after this analyze from being canceled by a
+ * delayed analyze report. Update shared stats entry directly for the
+ * above reasons.
+ */
+ tabentry = (PgStat_StatTabEntry *)
+ get_stat_entry(PGSTAT_TYPE_TABLE, dboid, RelationGetRelid(rel),
+ false, true, NULL);
+
+ LWLockAcquire(&tabentry->header.lock, LW_EXCLUSIVE);
+ tabentry->n_live_tuples = livetuples;
+ tabentry->n_dead_tuples = deadtuples;
+
+ /*
+ * If commanded, reset changes_since_analyze to zero. This forgets any
+ * changes that were committed while the ANALYZE was in progress, but we
+ * have no good way to estimate how many of those there were.
+ */
+ if (resetcounter)
+ tabentry->changes_since_analyze = 0;
+
+ if (IsAutoVacuumWorkerProcess())
+ {
+ tabentry->autovac_analyze_timestamp = GetCurrentTimestamp();
+ tabentry->autovac_analyze_count++;
+ }
+ else
+ {
+ tabentry->analyze_timestamp = GetCurrentTimestamp();
+ tabentry->analyze_count++;
+ }
+ LWLockRelease(&tabentry->header.lock);
}
/* --------
* pgstat_report_recovery_conflict() -
*
- * Tell the collector about a Hot Standby recovery conflict.
+ * Report a Hot Standby recovery conflict.
* --------
*/
void
pgstat_report_recovery_conflict(int reason)
{
- PgStat_MsgRecoveryConflict msg;
+ PgStat_StatDBEntry *dbent;
- if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ /* return if we are not collecting stats */
+ if (!area)
return;
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RECOVERYCONFLICT);
- msg.m_databaseid = MyDatabaseId;
- msg.m_reason = reason;
- pgstat_send(&msg, sizeof(msg));
+ dbent = get_local_dbstat_entry(MyDatabaseId);
+
+ switch (reason)
+ {
+ case PROCSIG_RECOVERY_CONFLICT_DATABASE:
+
+ /*
+ * Since we drop the information about the database as soon as it
+ * replicates, there is no point in counting these conflicts.
+ */
+ break;
+ case PROCSIG_RECOVERY_CONFLICT_TABLESPACE:
+ dbent->counts.n_conflict_tablespace++;
+ break;
+ case PROCSIG_RECOVERY_CONFLICT_LOCK:
+ dbent->counts.n_conflict_lock++;
+ break;
+ case PROCSIG_RECOVERY_CONFLICT_SNAPSHOT:
+ dbent->counts.n_conflict_snapshot++;
+ break;
+ case PROCSIG_RECOVERY_CONFLICT_BUFFERPIN:
+ dbent->counts.n_conflict_bufferpin++;
+ break;
+ case PROCSIG_RECOVERY_CONFLICT_STARTUP_DEADLOCK:
+ dbent->counts.n_conflict_startup_deadlock++;
+ break;
+ }
}
+
/* --------
* pgstat_report_deadlock() -
*
- * Tell the collector about a deadlock detected.
+ * Report a deadlock detected.
* --------
*/
void
pgstat_report_deadlock(void)
{
- PgStat_MsgDeadlock msg;
+ PgStat_StatDBEntry *dbent;
- if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ /* return if we are not collecting stats */
+ if (!area)
return;
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_DEADLOCK);
- msg.m_databaseid = MyDatabaseId;
- pgstat_send(&msg, sizeof(msg));
+ dbent = get_local_dbstat_entry(MyDatabaseId);
+ dbent->counts.n_deadlocks++;
}
-
-
/* --------
- * pgstat_report_checksum_failures_in_db() -
+ * pgstat_report_checksum_failures_in_db(dboid, failure_count) -
*
- * Tell the collector about one or more checksum failures.
+ * Reports about one or more checksum failures.
* --------
*/
void
pgstat_report_checksum_failures_in_db(Oid dboid, int failurecount)
{
- PgStat_MsgChecksumFailure msg;
+ PgStat_StatDBEntry *dbentry;
- if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ /* return if we are not active */
+ if (!area)
return;
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_CHECKSUMFAILURE);
- msg.m_databaseid = dboid;
- msg.m_failurecount = failurecount;
- msg.m_failure_time = GetCurrentTimestamp();
+ dbentry = get_local_dbstat_entry(dboid);
- pgstat_send(&msg, sizeof(msg));
+ /* add accumulated count to the parameter */
+ dbentry->counts.n_checksum_failures += failurecount;
}
/* --------
* pgstat_report_checksum_failure() -
*
- * Tell the collector about a checksum failure.
+ * Reports about a checksum failure.
* --------
*/
void
pgstat_report_checksum_failure(void)
{
- pgstat_report_checksum_failures_in_db(MyDatabaseId, 1);
+ PgStat_StatDBEntry *dbent;
+
+ /* return if we are not collecting stats */
+ if (!area)
+ return;
+
+ dbent = get_local_dbstat_entry(MyDatabaseId);
+ dbent->counts.n_checksum_failures++;
}
/* --------
* pgstat_report_tempfile() -
*
- * Tell the collector about a temporary file.
+ * Report a temporary file.
* --------
*/
void
pgstat_report_tempfile(size_t filesize)
{
- PgStat_MsgTempFile msg;
+ PgStat_StatDBEntry *dbent;
- if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ /* return if we are not collecting stats */
+ if (!area)
return;
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_TEMPFILE);
- msg.m_databaseid = MyDatabaseId;
- msg.m_filesize = filesize;
- pgstat_send(&msg, sizeof(msg));
+ if (filesize == 0) /* Is there a case where filesize is really 0? */
+ return;
+
+ dbent = get_local_dbstat_entry(MyDatabaseId);
+ dbent->counts.n_temp_bytes += filesize; /* needs check overflow */
+ dbent->counts.n_temp_files++;
}
/* ----------
* pgstat_report_replslot() -
*
- * Tell the collector about replication slot statistics.
+ * Report replication slot activity.
* ----------
*/
void
-pgstat_report_replslot(const char *slotname, int spilltxns, int spillcount,
- int spillbytes, int streamtxns, int streamcount, int streambytes)
+pgstat_report_replslot(const char *slotname,
+ int spilltxns, int spillcount, int spillbytes,
+ int streamtxns, int streamcount, int streambytes)
{
- PgStat_MsgReplSlot msg;
+ PgStat_ReplSlot *shent;
+ int i;
+ bool found;
+
+ if (!area)
+ return;
+
+ for (i = 0; i < max_replication_slots; i++)
+ {
+ if (strcmp(NameStr(ReplicationSlotCtl->replication_slots[i].data.name),
+ slotname) == 0)
+ break;
+
+ }
/*
- * Prepare and send the message
+ * the slot should have been removed. just ignore it. We create the entry
+ * for the slot with this name next time.
*/
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_REPLSLOT);
- strlcpy(msg.m_slotname, slotname, NAMEDATALEN);
- msg.m_drop = false;
- msg.m_spill_txns = spilltxns;
- msg.m_spill_count = spillcount;
- msg.m_spill_bytes = spillbytes;
- msg.m_stream_txns = streamtxns;
- msg.m_stream_count = streamcount;
- msg.m_stream_bytes = streambytes;
- pgstat_send(&msg, sizeof(PgStat_MsgReplSlot));
+ if (i == max_replication_slots)
+ return;
+
+ shent = (PgStat_ReplSlot *)
+ get_stat_entry(PGSTAT_TYPE_REPLSLOT,
+ MyDatabaseId, i, false, true, &found);
+
+ /* Clear the counters and reset dropped when we reuse it */
+ LWLockAcquire(&shent->header.lock, LW_EXCLUSIVE);
+ if (shent->header.dropped || !found)
+ {
+ memset(&shent->spill_txns, 0,
+ sizeof(PgStat_ReplSlot) - offsetof(PgStat_ReplSlot, spill_txns));
+ strlcpy(shent->slotname, slotname, NAMEDATALEN);
+ shent->header.dropped = false;
+ }
+
+ shent->spill_txns += spilltxns;
+ shent->spill_count += spillcount;
+ shent->spill_bytes += spillbytes;
+ shent->stream_txns += streamtxns;
+ shent->stream_count += streamcount;
+ shent->stream_bytes += streambytes;
+ LWLockRelease(&shent->header.lock);
}
/* ----------
@@ -1802,55 +2448,44 @@ pgstat_report_replslot(const char *slotname, int spilltxns, int spillcount,
void
pgstat_report_replslot_drop(const char *slotname)
{
- PgStat_MsgReplSlot msg;
+ int i;
+ PgStat_ReplSlot *shent;
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_REPLSLOT);
- strlcpy(msg.m_slotname, slotname, NAMEDATALEN);
- msg.m_drop = true;
- pgstat_send(&msg, sizeof(PgStat_MsgReplSlot));
-}
+ Assert(area);
+ if (!area)
+ return;
-/* ----------
- * pgstat_ping() -
- *
- * Send some junk data to the collector to increase traffic.
- * ----------
- */
-void
-pgstat_ping(void)
-{
- PgStat_MsgDummy msg;
+ for (i = 0; i < max_replication_slots; i++)
+ {
+ if (strcmp(NameStr(ReplicationSlotCtl->replication_slots[i].data.name),
+ slotname) == 0)
+ break;
+
+ }
- if (pgStatSock == PGINVALID_SOCKET)
+ /* XXX: maybe the slot has been removed. just ignore it. */
+ if (i == max_replication_slots)
return;
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_DUMMY);
- pgstat_send(&msg, sizeof(msg));
+ shent = (PgStat_ReplSlot *)
+ get_stat_entry(PGSTAT_TYPE_REPLSLOT,
+ MyDatabaseId, i, false, false, NULL);
+
+ if (shent && !shent->header.dropped)
+ {
+ LWLockAcquire(&shent->header.lock, LW_EXCLUSIVE);
+ shent->header.dropped = true;
+ LWLockRelease(&shent->header.lock);
+ }
}
/* ----------
- * pgstat_send_inquiry() -
+ * pgstat_init_function_usage() -
*
- * Notify collector that we need fresh data.
+ * Initialize function call usage data.
+ * Called by the executor before invoking a function.
* ----------
*/
-static void
-pgstat_send_inquiry(TimestampTz clock_time, TimestampTz cutoff_time, Oid databaseid)
-{
- PgStat_MsgInquiry msg;
-
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_INQUIRY);
- msg.clock_time = clock_time;
- msg.cutoff_time = cutoff_time;
- msg.databaseid = databaseid;
- pgstat_send(&msg, sizeof(msg));
-}
-
-
-/*
- * Initialize function call usage data.
- * Called by the executor before invoking a function.
- */
void
pgstat_init_function_usage(FunctionCallInfo fcinfo,
PgStat_FunctionCallUsage *fcu)
@@ -1865,24 +2500,9 @@ pgstat_init_function_usage(FunctionCallInfo fcinfo,
return;
}
- if (!pgStatFunctions)
- {
- /* First time through - initialize function stat table */
- HASHCTL hash_ctl;
-
- hash_ctl.keysize = sizeof(Oid);
- hash_ctl.entrysize = sizeof(PgStat_BackendFunctionEntry);
- pgStatFunctions = hash_create("Function stat entries",
- PGSTAT_FUNCTION_HASH_SIZE,
- &hash_ctl,
- HASH_ELEM | HASH_BLOBS);
- }
-
- /* Get the stats entry for this function, create if necessary */
- htabent = hash_search(pgStatFunctions, &fcinfo->flinfo->fn_oid,
- HASH_ENTER, &found);
- if (!found)
- MemSet(&htabent->f_counts, 0, sizeof(PgStat_FunctionCounts));
+ htabent = (PgStat_BackendFunctionEntry *)
+ get_local_stat_entry(PGSTAT_TYPE_FUNCTION, MyDatabaseId,
+ fcinfo->flinfo->fn_oid, true, &found);
fcu->fs = &htabent->f_counts;
@@ -1896,31 +2516,37 @@ pgstat_init_function_usage(FunctionCallInfo fcinfo,
INSTR_TIME_SET_CURRENT(fcu->f_start);
}
-/*
- * find_funcstat_entry - find any existing PgStat_BackendFunctionEntry entry
- * for specified function
+/* ----------
+ * find_funcstat_entry() -
*
- * If no entry, return NULL, don't create a new one
+ * find any existing PgStat_BackendFunctionEntry entry for specified function
+ *
+ * If no entry, return NULL, not creating a new one.
+ * ----------
*/
PgStat_BackendFunctionEntry *
find_funcstat_entry(Oid func_id)
{
- if (pgStatFunctions == NULL)
- return NULL;
+ PgStat_BackendFunctionEntry *ent;
- return (PgStat_BackendFunctionEntry *) hash_search(pgStatFunctions,
- (void *) &func_id,
- HASH_FIND, NULL);
+ ent = (PgStat_BackendFunctionEntry *)
+ get_local_stat_entry(PGSTAT_TYPE_FUNCTION, MyDatabaseId,
+ func_id, false, NULL);
+
+ return ent;
}
-/*
- * Calculate function call usage and update stat counters.
- * Called by the executor after invoking a function.
+/* ----------
+ * pgstat_end_function_usage() -
*
- * In the case of a set-returning function that runs in value-per-call mode,
- * we will see multiple pgstat_init_function_usage/pgstat_end_function_usage
- * calls for what the user considers a single call of the function. The
- * finalize flag should be TRUE on the last call.
+ * Calculate function call usage and update stat counters.
+ * Called by the executor after invoking a function.
+ *
+ * In the case of a set-returning function that runs in value-per-call mode,
+ * we will see multiple pgstat_init_function_usage/pgstat_end_function_usage
+ * calls for what the user considers a single call of the function. The
+ * finalize flag should be TRUE on the last call.
+ * ----------
*/
void
pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
@@ -1961,9 +2587,6 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
fs->f_numcalls++;
fs->f_total_time = f_total;
INSTR_TIME_ADD(fs->f_self_time, f_self);
-
- /* indicate that we have something to send */
- have_function_stats = true;
}
@@ -1975,8 +2598,7 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize)
*
* We assume that a relcache entry's pgstat_info field is zeroed by
* relcache.c when the relcache entry is made; thereafter it is long-lived
- * data. We can avoid repeated searches of the TabStatus arrays when the
- * same relation is touched repeatedly within a transaction.
+ * data.
* ----------
*/
void
@@ -1992,7 +2614,8 @@ pgstat_initstats(Relation rel)
return;
}
- if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts)
+ /* return if we are not collecting stats */
+ if (!area)
{
/* We're not counting at all */
rel->pgstat_info = NULL;
@@ -2003,120 +2626,60 @@ pgstat_initstats(Relation rel)
* If we already set up this relation in the current transaction, nothing
* to do.
*/
- if (rel->pgstat_info != NULL &&
- rel->pgstat_info->t_id == rel_id)
+ if (rel->pgstat_info != NULL)
return;
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = get_tabstat_entry(rel_id, rel->rd_rel->relisshared);
+ rel->pgstat_info = get_local_tabstat_entry(rel_id, rel->rd_rel->relisshared);
+ /* mark this relation as the owner */
+
+ /* don't allow link a stats to multiple relcache entries */
+ Assert(rel->pgstat_info->relation == NULL);
+ rel->pgstat_info->relation = rel;
}
/*
- * get_tabstat_entry - find or create a PgStat_TableStatus entry for rel
+ * pgstat_delinkstats() -
+ *
+ * Break the mutual link between a relcache entry and a local stats entry.
+ * This must be called always when one end of the link is removed.
*/
-static PgStat_TableStatus *
-get_tabstat_entry(Oid rel_id, bool isshared)
+void
+pgstat_delinkstats(Relation rel)
{
- TabStatHashEntry *hash_entry;
- PgStat_TableStatus *entry;
- TabStatusArray *tsa;
- bool found;
-
- /*
- * Create hash table if we don't have it already.
- */
- if (pgStatTabHash == NULL)
+ /* remove the link to stats info if any */
+ if (rel && rel->pgstat_info)
{
- HASHCTL ctl;
-
- ctl.keysize = sizeof(Oid);
- ctl.entrysize = sizeof(TabStatHashEntry);
-
- pgStatTabHash = hash_create("pgstat TabStatusArray lookup hash table",
- TABSTAT_QUANTUM,
- &ctl,
- HASH_ELEM | HASH_BLOBS);
+ /* ilnk sanity check */
+ Assert(rel->pgstat_info->relation == rel);
+ rel->pgstat_info->relation = NULL;
+ rel->pgstat_info = NULL;
}
-
- /*
- * Find an entry or create a new one.
- */
- hash_entry = hash_search(pgStatTabHash, &rel_id, HASH_ENTER, &found);
- if (!found)
- {
- /* initialize new entry with null pointer */
- hash_entry->tsa_entry = NULL;
- }
-
- /*
- * If entry is already valid, we're done.
- */
- if (hash_entry->tsa_entry)
- return hash_entry->tsa_entry;
-
- /*
- * Locate the first pgStatTabList entry with free space, making a new list
- * entry if needed. Note that we could get an OOM failure here, but if so
- * we have left the hashtable and the list in a consistent state.
- */
- if (pgStatTabList == NULL)
- {
- /* Set up first pgStatTabList entry */
- pgStatTabList = (TabStatusArray *)
- MemoryContextAllocZero(TopMemoryContext,
- sizeof(TabStatusArray));
- }
-
- tsa = pgStatTabList;
- while (tsa->tsa_used >= TABSTAT_QUANTUM)
- {
- if (tsa->tsa_next == NULL)
- tsa->tsa_next = (TabStatusArray *)
- MemoryContextAllocZero(TopMemoryContext,
- sizeof(TabStatusArray));
- tsa = tsa->tsa_next;
- }
-
- /*
- * Allocate a PgStat_TableStatus entry within this list entry. We assume
- * the entry was already zeroed, either at creation or after last use.
- */
- entry = &tsa->tsa_entries[tsa->tsa_used++];
- entry->t_id = rel_id;
- entry->t_shared = isshared;
-
- /*
- * Now we can fill the entry in pgStatTabHash.
- */
- hash_entry->tsa_entry = entry;
-
- return entry;
}
/*
* find_tabstat_entry - find any existing PgStat_TableStatus entry for rel
*
- * If no entry, return NULL, don't create a new one
+ * Find any existing PgStat_TableStatus entry for rel_id in the current
+ * database. If not found, try finding from shared tables.
*
- * Note: if we got an error in the most recent execution of pgstat_report_stat,
- * it's possible that an entry exists but there's no hashtable entry for it.
- * That's okay, we'll treat this case as "doesn't exist".
+ * If no entry found, return NULL, don't create a new one
+ * ----------
*/
PgStat_TableStatus *
find_tabstat_entry(Oid rel_id)
{
- TabStatHashEntry *hash_entry;
+ PgStat_TableStatus *ent;
- /* If hashtable doesn't exist, there are no entries at all */
- if (!pgStatTabHash)
- return NULL;
+ ent = (PgStat_TableStatus *)
+ get_local_stat_entry(PGSTAT_TYPE_TABLE, MyDatabaseId, rel_id,
+ false, NULL);
+ if (!ent)
+ ent = (PgStat_TableStatus *)
+ get_local_stat_entry(PGSTAT_TYPE_TABLE, InvalidOid, rel_id,
+ false, NULL);
- hash_entry = hash_search(pgStatTabHash, &rel_id, HASH_FIND, NULL);
- if (!hash_entry)
- return NULL;
-
- /* Note that this step could also return NULL, but that's correct */
- return hash_entry->tsa_entry;
+ return ent;
}
/*
@@ -2531,8 +3094,6 @@ AtPrepare_PgStat(void)
record.inserted_pre_trunc = trans->inserted_pre_trunc;
record.updated_pre_trunc = trans->updated_pre_trunc;
record.deleted_pre_trunc = trans->deleted_pre_trunc;
- record.t_id = tabstat->t_id;
- record.t_shared = tabstat->t_shared;
record.t_truncated = trans->truncated;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
@@ -2547,8 +3108,8 @@ AtPrepare_PgStat(void)
*
* All we need do here is unlink the transaction stats state from the
* nontransactional state. The nontransactional action counts will be
- * reported to the stats collector immediately, while the effects on live
- * and dead tuple counts are preserved in the 2PC state file.
+ * reported to the activity stats facility immediately, while the effects on
+ * live and dead tuple counts are preserved in the 2PC state file.
*
* Note: AtEOXact_PgStat is not called during PREPARE.
*/
@@ -2593,7 +3154,7 @@ pgstat_twophase_postcommit(TransactionId xid, uint16 info,
PgStat_TableStatus *pgstat_info;
/* Find or create a tabstat entry for the rel */
- pgstat_info = get_tabstat_entry(rec->t_id, rec->t_shared);
+ pgstat_info = get_local_tabstat_entry(rec->t_id, rec->t_shared);
/* Same math as in AtEOXact_PgStat, commit case */
pgstat_info->t_counts.t_tuples_inserted += rec->tuples_inserted;
@@ -2629,7 +3190,7 @@ pgstat_twophase_postabort(TransactionId xid, uint16 info,
PgStat_TableStatus *pgstat_info;
/* Find or create a tabstat entry for the rel */
- pgstat_info = get_tabstat_entry(rec->t_id, rec->t_shared);
+ pgstat_info = get_local_tabstat_entry(rec->t_id, rec->t_shared);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->t_truncated)
@@ -2649,85 +3210,138 @@ pgstat_twophase_postabort(TransactionId xid, uint16 info,
/* ----------
* pgstat_fetch_stat_dbentry() -
*
- * Support function for the SQL-callable pgstat* functions. Returns
- * the collected statistics for one database or NULL. NULL doesn't mean
- * that the database doesn't exist, it is just not yet known by the
- * collector, so the caller is better off to report ZERO instead.
+ * Find database stats entry on backends in a palloc'ed memory.
+ *
+ * The returned entry is stored in static memory so the content is valid until
+ * the next call of the same function for the different database.
* ----------
*/
PgStat_StatDBEntry *
pgstat_fetch_stat_dbentry(Oid dbid)
{
- /*
- * If not done for this transaction, read the statistics collector stats
- * file into some hash tables.
- */
- backend_read_statsfile();
-
- /*
- * Lookup the requested database; return NULL if not found
- */
- return (PgStat_StatDBEntry *) hash_search(pgStatDBHash,
- (void *) &dbid,
- HASH_FIND, NULL);
+ PgStat_StatDBEntry *shent;
+
+ /* should be called from backends */
+ Assert(IsUnderPostmaster);
+
+ /* the simple cache doesn't work properly for InvalidOid */
+ Assert(dbid != InvalidOid);
+
+ /* Return cached result if it is valid. */
+ if (cached_dbent_key.databaseid == dbid)
+ return &cached_dbent;
+
+ shent = (PgStat_StatDBEntry *)
+ get_stat_entry(PGSTAT_TYPE_DB, dbid, InvalidOid, true, false, NULL);
+
+ if (!shent)
+ return NULL;
+
+ LWLockAcquire(&shent->header.lock, LW_SHARED);
+ memcpy(&cached_dbent, shent, sizeof(PgStat_StatDBEntry));
+ LWLockRelease(&shent->header.lock);
+
+ /* remember the key for the cached entry */
+ cached_dbent_key.databaseid = dbid;
+
+ return &cached_dbent;
}
-
/* ----------
* pgstat_fetch_stat_tabentry() -
*
* Support function for the SQL-callable pgstat* functions. Returns
- * the collected statistics for one table or NULL. NULL doesn't mean
+ * the activity statistics for one table or NULL. NULL doesn't mean
* that the table doesn't exist, it is just not yet known by the
- * collector, so the caller is better off to report ZERO instead.
+ * activity statistics facilities, so the caller is better off to
+ * report ZERO instead.
* ----------
*/
PgStat_StatTabEntry *
pgstat_fetch_stat_tabentry(Oid relid)
{
- Oid dbid;
- PgStat_StatDBEntry *dbentry;
PgStat_StatTabEntry *tabentry;
- /*
- * If not done for this transaction, read the statistics collector stats
- * file into some hash tables.
- */
- backend_read_statsfile();
-
- /*
- * Lookup our database, then look in its table hash table.
- */
- dbid = MyDatabaseId;
- dbentry = (PgStat_StatDBEntry *) hash_search(pgStatDBHash,
- (void *) &dbid,
- HASH_FIND, NULL);
- if (dbentry != NULL && dbentry->tables != NULL)
- {
- tabentry = (PgStat_StatTabEntry *) hash_search(dbentry->tables,
- (void *) &relid,
- HASH_FIND, NULL);
- if (tabentry)
- return tabentry;
- }
+ tabentry = pgstat_fetch_stat_tabentry_extended(false, relid);
+ if (tabentry != NULL)
+ return tabentry;
/*
* If we didn't find it, maybe it's a shared table.
*/
- dbid = InvalidOid;
- dbentry = (PgStat_StatDBEntry *) hash_search(pgStatDBHash,
- (void *) &dbid,
- HASH_FIND, NULL);
- if (dbentry != NULL && dbentry->tables != NULL)
- {
- tabentry = (PgStat_StatTabEntry *) hash_search(dbentry->tables,
- (void *) &relid,
- HASH_FIND, NULL);
- if (tabentry)
- return tabentry;
- }
-
- return NULL;
+ tabentry = pgstat_fetch_stat_tabentry_extended(true, relid);
+ return tabentry;
+}
+
+
+/* ----------
+ * pgstat_fetch_stat_tabentry_extended() -
+ *
+ * Find table stats entry on backends in dbent. The returned entry is stored
+ * in static memory so the content is valid until the next call of the same
+ * function for the different table.
+ */
+PgStat_StatTabEntry *
+pgstat_fetch_stat_tabentry_extended(bool shared, Oid reloid)
+{
+ PgStat_StatTabEntry *shent;
+ Oid dboid = (shared ? InvalidOid : MyDatabaseId);
+
+ /* should be called from backends */
+ Assert(IsUnderPostmaster);
+
+ /* the simple cache doesn't work properly for the InvalidOid */
+ Assert(reloid != InvalidOid);
+
+ /* Return cached result if it is valid. */
+ if (cached_tabent_key.databaseid == dboid &&
+ cached_tabent_key.objectid == reloid)
+ return &cached_tabent;
+
+ shent = (PgStat_StatTabEntry *)
+ get_stat_entry(PGSTAT_TYPE_TABLE, dboid, reloid, true, false, NULL);
+
+ if (!shent)
+ return NULL;
+
+ LWLockAcquire(&shent->header.lock, LW_SHARED);
+ memcpy(&cached_tabent, shent, sizeof(PgStat_StatTabEntry));
+ LWLockRelease(&shent->header.lock);
+
+ /* remember the key for the cached entry */
+ cached_tabent_key.databaseid = dboid;
+ cached_tabent_key.objectid = reloid;
+
+ return &cached_tabent;
+}
+
+
+/* ----------
+ * pgstat_copy_index_counters() -
+ *
+ * Support function for index swapping. Copy a portion of the counters of the
+ * relation to specified place.
+ * ----------
+ */
+void
+pgstat_copy_index_counters(Oid relid, PgStat_TableStatus *dst)
+{
+ PgStat_StatTabEntry *tabentry;
+
+ /* No point fetching tabentry when dst is NULL */
+ if (!dst)
+ return;
+
+ tabentry = pgstat_fetch_stat_tabentry(relid);
+
+ if (!tabentry)
+ return;
+
+ dst->t_counts.t_numscans = tabentry->numscans;
+ dst->t_counts.t_tuples_returned = tabentry->tuples_returned;
+ dst->t_counts.t_tuples_fetched = tabentry->tuples_fetched;
+ dst->t_counts.t_blocks_fetched = tabentry->blocks_fetched;
+ dst->t_counts.t_blocks_hit = tabentry->blocks_hit;
}
@@ -2736,30 +3350,46 @@ pgstat_fetch_stat_tabentry(Oid relid)
*
* Support function for the SQL-callable pgstat* functions. Returns
* the collected statistics for one function or NULL.
+ *
+ * The returned entry is stored in static memory so the content is valid until
+ * the next call of the same function for the different function id.
* ----------
*/
PgStat_StatFuncEntry *
pgstat_fetch_stat_funcentry(Oid func_id)
{
- PgStat_StatDBEntry *dbentry;
- PgStat_StatFuncEntry *funcentry = NULL;
-
- /* load the stats file if needed */
- backend_read_statsfile();
-
- /* Lookup our database, then find the requested function. */
- dbentry = pgstat_fetch_stat_dbentry(MyDatabaseId);
- if (dbentry != NULL && dbentry->functions != NULL)
- {
- funcentry = (PgStat_StatFuncEntry *) hash_search(dbentry->functions,
- (void *) &func_id,
- HASH_FIND, NULL);
- }
-
- return funcentry;
+ PgStat_StatFuncEntry *shent;
+ Oid dboid = MyDatabaseId;
+
+ /* should be called from backends */
+ Assert(IsUnderPostmaster);
+
+ /* the simple cache doesn't work properly for the InvalidOid */
+ Assert(func_id != InvalidOid);
+
+ /* Return cached result if it is valid. */
+ if (cached_funcent_key.databaseid == dboid &&
+ cached_funcent_key.objectid == func_id)
+ return &cached_funcent;
+
+ shent = (PgStat_StatFuncEntry *)
+ get_stat_entry(PGSTAT_TYPE_FUNCTION, dboid, func_id, true, false,
+ NULL);
+
+ if (!shent)
+ return NULL;
+
+ LWLockAcquire(&shent->header.lock, LW_SHARED);
+ memcpy(&cached_funcent, shent, sizeof(PgStat_StatFuncEntry));
+ LWLockRelease(&shent->header.lock);
+
+ /* remember the key for the cached entry */
+ cached_funcent_key.databaseid = dboid;
+ cached_funcent_key.objectid = func_id;
+
+ return &cached_funcent;
}
-
/* ----------
* pgstat_fetch_stat_beentry() -
*
@@ -2819,53 +3449,160 @@ pgstat_fetch_stat_numbackends(void)
return localNumBackends;
}
+/*
+ * ---------
+ * pgstat_get_stat_timestamp() -
+ *
+ * Returns the last update timstamp of global staticstics.
+ */
+TimestampTz
+pgstat_get_stat_timestamp(void)
+{
+ return (TimestampTz) pg_atomic_read_u64(&StatsShmem->stats_timestamp);
+}
+
/*
* ---------
* pgstat_fetch_stat_archiver() -
*
- * Support function for the SQL-callable pgstat* functions. Returns
- * a pointer to the archiver statistics struct.
+ * Support function for the SQL-callable pgstat* functions. The returned
+ * entry is stored in static memory so the content is valid until the next
+ * call.
* ---------
*/
-PgStat_ArchiverStats *
+PgStat_Archiver *
pgstat_fetch_stat_archiver(void)
{
- backend_read_statsfile();
+ PgStat_Archiver reset;
+ PgStat_Archiver *reset_shared = &StatsShmem->archiver_reset_offset;
+ PgStat_Archiver *shared = &StatsShmem->archiver_stats;
+ PgStat_Archiver *cached = &cached_archiverstats;
- return &archiverStats;
+ pgstat_copy_global_stats(cached, shared, sizeof(PgStat_Archiver),
+ &StatsShmem->archiver_changecount);
+
+ LWLockAcquire(StatsLock, LW_SHARED);
+ memcpy(&reset, reset_shared, sizeof(PgStat_Archiver));
+ LWLockRelease(StatsLock);
+
+ /* compensate by reset offsets */
+ if (cached->archived_count == reset.archived_count)
+ {
+ cached->last_archived_wal[0] = 0;
+ cached->last_archived_timestamp = 0;
+ }
+ cached->archived_count -= reset.archived_count;
+
+ if (cached->failed_count == reset.failed_count)
+ {
+ cached->last_failed_wal[0] = 0;
+ cached->last_failed_timestamp = 0;
+ }
+ cached->failed_count -= reset.failed_count;
+
+ cached->stat_reset_timestamp = reset.stat_reset_timestamp;
+
+ cached_archiverstats_is_valid = true;
+
+ return &cached_archiverstats;
}
/*
* ---------
- * pgstat_fetch_global() -
+ * pgstat_fetch_stat_bgwriter() -
*
- * Support function for the SQL-callable pgstat* functions. Returns
- * a pointer to the global statistics struct.
+ * Support function for the SQL-callable pgstat* functions. The returned
+ * entry is stored in static memory so the content is valid until the next
+ * call.
* ---------
*/
-PgStat_GlobalStats *
-pgstat_fetch_global(void)
+PgStat_BgWriter *
+pgstat_fetch_stat_bgwriter(void)
{
- backend_read_statsfile();
+ PgStat_BgWriter reset;
+ PgStat_BgWriter *reset_shared = &StatsShmem->bgwriter_reset_offset;
+ PgStat_BgWriter *shared = &StatsShmem->bgwriter_stats;
+ PgStat_BgWriter *cached = &cached_bgwriterstats;
+
+ pgstat_copy_global_stats(cached, shared, sizeof(PgStat_BgWriter),
+ &StatsShmem->bgwriter_changecount);
+
+ LWLockAcquire(StatsLock, LW_SHARED);
+ memcpy(&reset, reset_shared, sizeof(PgStat_BgWriter));
+ LWLockRelease(StatsLock);
+
+ /* compensate by reset offsets */
+ cached->buf_written_clean -= reset.buf_written_clean;
+ cached->maxwritten_clean -= reset.maxwritten_clean;
+ cached->buf_alloc -= reset.buf_alloc;
+ cached->stat_reset_timestamp = reset.stat_reset_timestamp;
+
+ cached_bgwriterstats_is_valid = true;
+
+ return &cached_bgwriterstats;
+}
+
+/*
+ * ---------
+ * pgstat_fetch_stat_checkpinter() -
+ *
+ * Support function for the SQL-callable pgstat* functions. The returned
+ * entry is stored in static memory so the content is valid until the next
+ * call.
+ * ---------
+ */
+PgStat_CheckPointer *
+pgstat_fetch_stat_checkpointer(void)
+{
+ PgStat_CheckPointer reset;
+ PgStat_CheckPointer *reset_shared = &StatsShmem->checkpointer_reset_offset;
+ PgStat_CheckPointer *shared = &StatsShmem->checkpointer_stats;
+ PgStat_CheckPointer *cached = &cached_checkpointerstats;
+
+ pgstat_copy_global_stats(cached, shared, sizeof(PgStat_CheckPointer),
+ &StatsShmem->checkpointer_changecount);
+
+ LWLockAcquire(StatsLock, LW_SHARED);
+ memcpy(&reset, reset_shared, sizeof(PgStat_CheckPointer));
+ LWLockRelease(StatsLock);
+
+ /* compensate by reset offsets */
+ cached->timed_checkpoints -= reset.timed_checkpoints;
+ cached->requested_checkpoints -= reset.requested_checkpoints;
+ cached->buf_written_checkpoints -= reset.buf_written_checkpoints;
+ cached->buf_written_backend -= reset.buf_written_backend;
+ cached->buf_fsync_backend -= reset.buf_fsync_backend;
+ cached->checkpoint_write_time -= reset.checkpoint_write_time;
+ cached->checkpoint_sync_time -= reset.checkpoint_sync_time;
+
+ cached_checkpointerstats_is_valid = true;
- return &globalStats;
+ return &cached_checkpointerstats;
}
/*
* ---------
* pgstat_fetch_stat_wal() -
*
- * Support function for the SQL-callable pgstat* functions. Returns
- * a pointer to the WAL statistics struct.
+ * Support function for the SQL-callable pgstat* functions. The returned entry
+ * is stored in static memory so the content is valid until the next
+ * call.
* ---------
*/
-PgStat_WalStats *
+PgStat_Wal *
pgstat_fetch_stat_wal(void)
{
- backend_read_statsfile();
+ if (!cached_walstats_is_valid)
+ {
+ LWLockAcquire(StatsLock, LW_SHARED);
+ memcpy(&cached_walstats, &StatsShmem->wal_stats, sizeof(PgStat_Wal));
+ LWLockRelease(StatsLock);
+ }
- return &walStats;
+ cached_walstats_is_valid = true;
+
+ return &cached_walstats;
}
/*
@@ -2879,9 +3616,27 @@ pgstat_fetch_stat_wal(void)
PgStat_SLRUStats *
pgstat_fetch_slru(void)
{
- backend_read_statsfile();
+ size_t size = sizeof(PgStat_SLRUStats) * SLRU_NUM_ELEMENTS;
- return slruStats;
+ for (;;)
+ {
+ uint32 before_count;
+ uint32 after_count;
+
+ pg_read_barrier();
+ before_count = pg_atomic_read_u32(&StatsShmem->slru_changecount);
+ memcpy(&cached_slrustats, &StatsShmem->slru_stats, size);
+ after_count = pg_atomic_read_u32(&StatsShmem->slru_changecount);
+
+ if (before_count == after_count && (before_count & 1) == 0)
+ break;
+
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ cached_slrustats_is_valid = true;
+
+ return &cached_slrustats;
}
/*
@@ -2893,13 +3648,42 @@ pgstat_fetch_slru(void)
* number of entries in nslots_p.
* ---------
*/
-PgStat_ReplSlotStats *
+PgStat_ReplSlot *
pgstat_fetch_replslot(int *nslots_p)
{
- backend_read_statsfile();
- *nslots_p = nReplSlotStats;
- return replSlotStats;
+ if (cached_replslotstats == NULL)
+ {
+ cached_replslotstats = (PgStat_ReplSlot *)
+ MemoryContextAlloc(pgStatCacheContext,
+ sizeof(PgStat_ReplSlot) * max_replication_slots);
+ }
+
+ if (n_cached_replslotstats < 0)
+ {
+ int n = 0;
+ int i;
+
+ for (i = 0; i < max_replication_slots; i++)
+ {
+ PgStat_ReplSlot *shent = (PgStat_ReplSlot *)
+ get_stat_entry(PGSTAT_TYPE_REPLSLOT, MyDatabaseId, i,
+ false, false, NULL);
+
+ if (shent && !shent->header.dropped)
+ {
+ memcpy(cached_replslotstats[n++].slotname,
+ shent->slotname,
+ sizeof(PgStat_ReplSlot) -
+ offsetof(PgStat_ReplSlot, slotname));
+ }
+ }
+
+ n_cached_replslotstats = n;
+ }
+
+ *nslots_p = n_cached_replslotstats;
+ return cached_replslotstats;
}
/* ------------------------------------------------------------
@@ -3124,8 +3908,8 @@ pgstat_initialize(void)
*/
prevWalUsage = pgWalUsage;
- /* Set up a process-exit hook to clean up */
- on_shmem_exit(pgstat_beshutdown_hook, 0);
+ /* need to be called before dsm shutdown */
+ before_shmem_exit(pgstat_beshutdown_hook, 0);
}
/* ----------
@@ -3301,12 +4085,15 @@ pgstat_bestart(void)
/* Update app name to current GUC setting */
if (application_name)
pgstat_report_appname(application_name);
+
+ /* attach shared database stats area */
+ attach_shared_stats();
}
/*
* Shut down a single backend's statistics reporting at process exit.
*
- * Flush any remaining statistics counts out to the collector.
+ * Flush any remaining statistics counts out to shared stats.
* Without this, operations triggered during backend exit (such as
* temp table deletions) won't be counted.
*
@@ -3319,12 +4106,25 @@ pgstat_beshutdown_hook(int code, Datum arg)
/*
* If we got as far as discovering our own database ID, we can report what
- * we did to the collector. Otherwise, we'd be sending an invalid
+ * we did to the shares stats. Otherwise, we'd be sending an invalid
* database ID, so forget it. (This means that accesses to pg_database
* during failed backend starts might never get counted.)
*/
if (OidIsValid(MyDatabaseId))
+ {
+ if (MyBackendType == B_BACKEND)
+ pgstat_update_connstats(true);
pgstat_report_stat(true);
+ }
+
+ /*
+ * We need to clean up temporary slots before detaching shared statistics
+ * so that the statistics for temporary slots are properly removed.
+ */
+ if (MyReplicationSlot != NULL)
+ ReplicationSlotRelease();
+
+ ReplicationSlotCleanup();
/*
* Clear my status entry, following the protocol of bumping st_changecount
@@ -3336,6 +4136,8 @@ pgstat_beshutdown_hook(int code, Datum arg)
beentry->st_procpid = 0; /* mark invalid */
PGSTAT_END_WRITE_ACTIVITY(beentry);
+
+ detach_shared_stats(true);
}
@@ -3620,7 +4422,8 @@ pgstat_read_current_status(void)
#endif
int i;
- Assert(!pgStatRunningInCollector);
+ Assert(IsUnderPostmaster);
+
if (localBackendStatusTable)
return; /* already done */
@@ -3915,8 +4718,8 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_LOGICAL_LAUNCHER_MAIN:
event_name = "LogicalLauncherMain";
break;
- case WAIT_EVENT_PGSTAT_MAIN:
- event_name = "PgStatMain";
+ case WAIT_EVENT_READING_STATS_FILE:
+ event_name = "ReadingStatsFile";
break;
case WAIT_EVENT_RECOVERY_WAL_STREAM:
event_name = "RecoveryWalStream";
@@ -4570,94 +5373,80 @@ pgstat_get_crashed_backend_activity(int pid, char *buffer, int buflen)
/* ----------
- * pgstat_setheader() -
+ * pgstat_report_archiver() -
*
- * Set common header fields in a statistics message
+ * Report archiver statistics
* ----------
*/
-static void
-pgstat_setheader(PgStat_MsgHdr *hdr, StatMsgType mtype)
-{
- hdr->m_type = mtype;
-}
-
-
-/* ----------
- * pgstat_send() -
- *
- * Send out one statistics message to the collector
- * ----------
- */
-static void
-pgstat_send(void *msg, int len)
+void
+pgstat_report_archiver(const char *xlog, bool failed)
{
- int rc;
+ TimestampTz now = GetCurrentTimestamp();
+ uint32 before_count PG_USED_FOR_ASSERTS_ONLY;
+ uint32 after_count PG_USED_FOR_ASSERTS_ONLY;
- if (pgStatSock == PGINVALID_SOCKET)
- return;
- ((PgStat_MsgHdr *) msg)->m_size = len;
+ START_CRIT_SECTION();
+ before_count =
+ pg_atomic_fetch_add_u32(&StatsShmem->archiver_changecount, 1);
+ Assert((before_count & 1) == 0);
- /* We'll retry after EINTR, but ignore all other failures */
- do
+ if (failed)
{
- rc = send(pgStatSock, msg, len, 0);
- } while (rc < 0 && errno == EINTR);
-
-#ifdef USE_ASSERT_CHECKING
- /* In debug builds, log send failures ... */
- if (rc < 0)
- elog(LOG, "could not send to statistics collector: %m");
-#endif
-}
-
-/* ----------
- * pgstat_send_archiver() -
- *
- * Tell the collector about the WAL file that we successfully
- * archived or failed to archive.
- * ----------
- */
-void
-pgstat_send_archiver(const char *xlog, bool failed)
-{
- PgStat_MsgArchiver msg;
-
- /*
- * Prepare and send the message
- */
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_ARCHIVER);
- msg.m_failed = failed;
- strlcpy(msg.m_xlog, xlog, sizeof(msg.m_xlog));
- msg.m_timestamp = GetCurrentTimestamp();
- pgstat_send(&msg, sizeof(msg));
+ ++StatsShmem->archiver_stats.failed_count;
+ memcpy(&StatsShmem->archiver_stats.last_failed_wal, xlog,
+ sizeof(StatsShmem->archiver_stats.last_failed_wal));
+ StatsShmem->archiver_stats.last_failed_timestamp = now;
+ }
+ else
+ {
+ ++StatsShmem->archiver_stats.archived_count;
+ memcpy(&StatsShmem->archiver_stats.last_archived_wal, xlog,
+ sizeof(StatsShmem->archiver_stats.last_archived_wal));
+ StatsShmem->archiver_stats.last_archived_timestamp = now;
+ }
+
+ after_count =
+ pg_atomic_fetch_add_u32(&StatsShmem->archiver_changecount, 1);
+ Assert(after_count == before_count + 1);
+ END_CRIT_SECTION();
}
/* ----------
- * pgstat_send_bgwriter() -
+ * pgstat_report_bgwriter() -
*
- * Send bgwriter statistics to the collector
+ * Report bgwriter statistics
* ----------
*/
void
-pgstat_send_bgwriter(void)
+pgstat_report_bgwriter(void)
{
- /* We assume this initializes to zeroes */
- static const PgStat_MsgBgWriter all_zeroes;
+ static const PgStat_BgWriter all_zeroes;
+ PgStat_BgWriter *s = &StatsShmem->bgwriter_stats;
+ PgStat_BgWriter *l = &BgWriterStats;
+ uint32 before_count PG_USED_FOR_ASSERTS_ONLY;
+ uint32 after_count PG_USED_FOR_ASSERTS_ONLY;
/*
* This function can be called even if nothing at all has happened. In
- * this case, avoid sending a completely empty message to the stats
- * collector.
+ * this case, avoid taking lock for a completely empty stats.
*/
- if (memcmp(&BgWriterStats, &all_zeroes, sizeof(PgStat_MsgBgWriter)) == 0)
+ if (memcmp(&BgWriterStats, &all_zeroes, sizeof(PgStat_BgWriter)) == 0)
return;
- /*
- * Prepare and send the message
- */
- pgstat_setheader(&BgWriterStats.m_hdr, PGSTAT_MTYPE_BGWRITER);
- pgstat_send(&BgWriterStats, sizeof(BgWriterStats));
+ START_CRIT_SECTION();
+ before_count =
+ pg_atomic_fetch_add_u32(&StatsShmem->bgwriter_changecount, 1);
+ Assert((before_count & 1) == 0);
+
+ s->buf_written_clean += l->buf_written_clean;
+ s->maxwritten_clean += l->maxwritten_clean;
+ s->buf_alloc += l->buf_alloc;
+
+ after_count =
+ pg_atomic_fetch_add_u32(&StatsShmem->bgwriter_changecount, 1);
+ Assert(after_count == before_count + 1);
+ END_CRIT_SECTION();
/*
* Clear out the statistics buffer, so it can be re-used.
@@ -4665,589 +5454,194 @@ pgstat_send_bgwriter(void)
MemSet(&BgWriterStats, 0, sizeof(BgWriterStats));
}
+/* ----------
+ * pgstat_report_checkpointer() -
+ *
+ * Report checkpointer statistics
+ * ----------
+ */
+void
+pgstat_report_checkpointer(void)
+{
+ /* We assume this initializes to zeroes */
+ static const PgStat_CheckPointer all_zeroes;
+ PgStat_CheckPointer *s = &StatsShmem->checkpointer_stats;
+ PgStat_CheckPointer *l = &CheckPointerStats;
+ uint32 before_count PG_USED_FOR_ASSERTS_ONLY;
+ uint32 after_count PG_USED_FOR_ASSERTS_ONLY;
+
+ /*
+ * This function can be called even if nothing at all has happened. In
+ * this case, avoid taking lock for a completely empty stats.
+ */
+ if (memcmp(&CheckPointerStats, &all_zeroes,
+ sizeof(PgStat_CheckPointer)) == 0)
+ return;
+
+ START_CRIT_SECTION();
+ before_count =
+ pg_atomic_fetch_add_u32(&StatsShmem->checkpointer_changecount, 1);
+ Assert((before_count & 1) == 0);
+
+ s->timed_checkpoints += l->timed_checkpoints;
+ s->requested_checkpoints += l->requested_checkpoints;
+ s->checkpoint_write_time += l->checkpoint_write_time;
+ s->checkpoint_sync_time += l->checkpoint_sync_time;
+ s->buf_written_checkpoints += l->buf_written_checkpoints;
+ s->buf_written_backend += l->buf_written_backend;
+ s->buf_fsync_backend += l->buf_fsync_backend;
+
+ after_count =
+ pg_atomic_fetch_add_u32(&StatsShmem->checkpointer_changecount, 1);
+ Assert(after_count == before_count + 1);
+ END_CRIT_SECTION();
+
+ /*
+ * Clear out the statistics buffer, so it can be re-used.
+ */
+ MemSet(&CheckPointerStats, 0, sizeof(CheckPointerStats));
+}
+
/* ----------
* pgstat_report_wal() -
*
- * Calculate how much WAL usage counters are increased and send
- * WAL statistics to the collector.
- *
- * Must be called by processes that generate WAL.
+ * Report WAL statistics
* ----------
*/
void
pgstat_report_wal(void)
{
- WalUsage walusage;
+ flush_walstat(false);
+}
- /*
- * Calculate how much WAL usage counters are increased by substracting the
- * previous counters from the current ones. Fill the results in WAL stats
- * message.
- */
- MemSet(&walusage, 0, sizeof(WalUsage));
- WalUsageAccumDiff(&walusage, &pgWalUsage, &prevWalUsage);
+/* ----------
+ * pgstat_update_connstat() -
+ *
+ * Update local connection stats
+ * ----------
+ */
+static void
+pgstat_update_connstats(bool disconnect)
+{
+ static TimestampTz last_report = 0;
+ static SessionEndType session_end_type = DISCONNECT_NOT_YET;
+ TimestampTz now;
+ long secs;
+ int usecs;
+ PgStat_StatDBEntry *ldbstats; /* local database entry */
- WalStats.m_wal_records = walusage.wal_records;
- WalStats.m_wal_fpi = walusage.wal_fpi;
- WalStats.m_wal_bytes = walusage.wal_bytes;
+ Assert(MyBackendType == B_BACKEND);
- /*
- * Send WAL stats message to the collector.
- */
- if (!pgstat_send_wal(true))
+ if (session_end_type != DISCONNECT_NOT_YET)
return;
- /*
- * Save the current counters for the subsequent calculation of WAL usage.
- */
- prevWalUsage = pgWalUsage;
-}
+ now = GetCurrentTimestamp();
+ if (last_report == 0)
+ last_report = MyStartTimestamp;
+ TimestampDifference(last_report, now, &secs, &usecs);
+ last_report = now;
-/* ----------
- * pgstat_send_wal() -
- *
- * Send WAL statistics to the collector.
- *
- * If 'force' is not set, WAL stats message is only sent if enough time has
- * passed since last one was sent to reach PGSTAT_STAT_INTERVAL.
- *
- * Return true if the message is sent, and false otherwise.
- * ----------
- */
-bool
-pgstat_send_wal(bool force)
-{
- /* We assume this initializes to zeroes */
- static const PgStat_MsgWal all_zeroes;
- static TimestampTz sendTime = 0;
+ if (disconnect)
+ session_end_type = pgStatSessionEndCause;
- /*
- * This function can be called even if nothing at all has happened. In
- * this case, avoid sending a completely empty message to the stats
- * collector.
- */
- if (memcmp(&WalStats, &all_zeroes, sizeof(PgStat_MsgWal)) == 0)
- return false;
+ ldbstats = get_local_dbstat_entry(MyDatabaseId);
- if (!force)
- {
- TimestampTz now = GetCurrentTimestamp();
+ ldbstats->counts.n_sessions = (last_report == 0 ? 1 : 0);
+ ldbstats->counts.total_session_time += secs * 1000000 + usecs;
+ ldbstats->counts.total_active_time += pgStatActiveTime;
+ pgStatActiveTime = 0;
+ ldbstats->counts.total_idle_in_xact_time += pgStatTransactionIdleTime;
+ pgStatTransactionIdleTime = 0;
- /*
- * Don't send a message unless it's been at least PGSTAT_STAT_INTERVAL
- * msec since we last sent one.
- */
- if (!TimestampDifferenceExceeds(sendTime, now, PGSTAT_STAT_INTERVAL))
- return false;
- sendTime = now;
- }
-
- /*
- * Prepare and send the message
- */
- pgstat_setheader(&WalStats.m_hdr, PGSTAT_MTYPE_WAL);
- pgstat_send(&WalStats, sizeof(WalStats));
-
- /*
- * Clear out the statistics buffer, so it can be re-used.
- */
- MemSet(&WalStats, 0, sizeof(WalStats));
-
- return true;
-}
-
-/* ----------
- * pgstat_send_slru() -
- *
- * Send SLRU statistics to the collector
- * ----------
- */
-static void
-pgstat_send_slru(void)
-{
- /* We assume this initializes to zeroes */
- static const PgStat_MsgSLRU all_zeroes;
-
- for (int i = 0; i < SLRU_NUM_ELEMENTS; i++)
+ switch (session_end_type)
{
- /*
- * This function can be called even if nothing at all has happened. In
- * this case, avoid sending a completely empty message to the stats
- * collector.
- */
- if (memcmp(&SLRUStats[i], &all_zeroes, sizeof(PgStat_MsgSLRU)) == 0)
- continue;
-
- /* set the SLRU type before each send */
- SLRUStats[i].m_index = i;
-
- /*
- * Prepare and send the message
- */
- pgstat_setheader(&SLRUStats[i].m_hdr, PGSTAT_MTYPE_SLRU);
- pgstat_send(&SLRUStats[i], sizeof(PgStat_MsgSLRU));
-
- /*
- * Clear out the statistics buffer, so it can be re-used.
- */
- MemSet(&SLRUStats[i], 0, sizeof(PgStat_MsgSLRU));
- }
-}
-
-
-/* ----------
- * PgstatCollectorMain() -
- *
- * Start up the statistics collector process. This is the body of the
- * postmaster child process.
- *
- * The argc/argv parameters are valid only in EXEC_BACKEND case.
- * ----------
- */
-NON_EXEC_STATIC void
-PgstatCollectorMain(int argc, char *argv[])
-{
- int len;
- PgStat_Msg msg;
- int wr;
- WaitEvent event;
- WaitEventSet *wes;
-
- /*
- * Ignore all signals usually bound to some action in the postmaster,
- * except SIGHUP and SIGQUIT. Note we don't need a SIGUSR1 handler to
- * support latch operations, because we only use a local latch.
- */
- pqsignal(SIGHUP, SignalHandlerForConfigReload);
- pqsignal(SIGINT, SIG_IGN);
- pqsignal(SIGTERM, SIG_IGN);
- pqsignal(SIGQUIT, SignalHandlerForShutdownRequest);
- pqsignal(SIGALRM, SIG_IGN);
- pqsignal(SIGPIPE, SIG_IGN);
- pqsignal(SIGUSR1, SIG_IGN);
- pqsignal(SIGUSR2, SIG_IGN);
- /* Reset some signals that are accepted by postmaster but not here */
- pqsignal(SIGCHLD, SIG_DFL);
- PG_SETMASK(&UnBlockSig);
-
- MyBackendType = B_STATS_COLLECTOR;
- init_ps_display(NULL);
-
- /*
- * Read in existing stats files or initialize the stats to zero.
- */
- pgStatRunningInCollector = true;
- pgStatDBHash = pgstat_read_statsfiles(InvalidOid, true, true);
-
- /* Prepare to wait for our latch or data in our socket. */
- wes = CreateWaitEventSet(CurrentMemoryContext, 3);
- AddWaitEventToSet(wes, WL_LATCH_SET, PGINVALID_SOCKET, MyLatch, NULL);
- AddWaitEventToSet(wes, WL_POSTMASTER_DEATH, PGINVALID_SOCKET, NULL, NULL);
- AddWaitEventToSet(wes, WL_SOCKET_READABLE, pgStatSock, NULL, NULL);
-
- /*
- * Loop to process messages until we get SIGQUIT or detect ungraceful
- * death of our parent postmaster.
- *
- * For performance reasons, we don't want to do ResetLatch/WaitLatch after
- * every message; instead, do that only after a recv() fails to obtain a
- * message. (This effectively means that if backends are sending us stuff
- * like mad, we won't notice postmaster death until things slack off a
- * bit; which seems fine.) To do that, we have an inner loop that
- * iterates as long as recv() succeeds. We do check ConfigReloadPending
- * inside the inner loop, which means that such interrupts will get
- * serviced but the latch won't get cleared until next time there is a
- * break in the action.
- */
- for (;;)
- {
- /* Clear any already-pending wakeups */
- ResetLatch(MyLatch);
-
- /*
- * Quit if we get SIGQUIT from the postmaster.
- */
- if (ShutdownRequestPending)
+ case DISCONNECT_NOT_YET:
+ case DISCONNECT_NORMAL:
+ /* we don't collect these */
break;
-
- /*
- * Inner loop iterates as long as we keep getting messages, or until
- * ShutdownRequestPending becomes set.
- */
- while (!ShutdownRequestPending)
- {
- /*
- * Reload configuration if we got SIGHUP from the postmaster.
- */
- if (ConfigReloadPending)
- {
- ConfigReloadPending = false;
- ProcessConfigFile(PGC_SIGHUP);
- }
-
- /*
- * Write the stats file(s) if a new request has arrived that is
- * not satisfied by existing file(s).
- */
- if (pgstat_write_statsfile_needed())
- pgstat_write_statsfiles(false, false);
-
- /*
- * Try to receive and process a message. This will not block,
- * since the socket is set to non-blocking mode.
- *
- * XXX On Windows, we have to force pgwin32_recv to cooperate,
- * despite the previous use of pg_set_noblock() on the socket.
- * This is extremely broken and should be fixed someday.
- */
-#ifdef WIN32
- pgwin32_noblock = 1;
-#endif
-
- len = recv(pgStatSock, (char *) &msg,
- sizeof(PgStat_Msg), 0);
-
-#ifdef WIN32
- pgwin32_noblock = 0;
-#endif
-
- if (len < 0)
- {
- if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
- break; /* out of inner loop */
- ereport(ERROR,
- (errcode_for_socket_access(),
- errmsg("could not read statistics message: %m")));
- }
-
- /*
- * We ignore messages that are smaller than our common header
- */
- if (len < sizeof(PgStat_MsgHdr))
- continue;
-
- /*
- * The received length must match the length in the header
- */
- if (msg.msg_hdr.m_size != len)
- continue;
-
- /*
- * O.K. - we accept this message. Process it.
- */
- switch (msg.msg_hdr.m_type)
- {
- case PGSTAT_MTYPE_DUMMY:
- break;
-
- case PGSTAT_MTYPE_INQUIRY:
- pgstat_recv_inquiry(&msg.msg_inquiry, len);
- break;
-
- case PGSTAT_MTYPE_TABSTAT:
- pgstat_recv_tabstat(&msg.msg_tabstat, len);
- break;
-
- case PGSTAT_MTYPE_TABPURGE:
- pgstat_recv_tabpurge(&msg.msg_tabpurge, len);
- break;
-
- case PGSTAT_MTYPE_DROPDB:
- pgstat_recv_dropdb(&msg.msg_dropdb, len);
- break;
-
- case PGSTAT_MTYPE_RESETCOUNTER:
- pgstat_recv_resetcounter(&msg.msg_resetcounter, len);
- break;
-
- case PGSTAT_MTYPE_RESETSHAREDCOUNTER:
- pgstat_recv_resetsharedcounter(&msg.msg_resetsharedcounter,
- len);
- break;
-
- case PGSTAT_MTYPE_RESETSINGLECOUNTER:
- pgstat_recv_resetsinglecounter(&msg.msg_resetsinglecounter,
- len);
- break;
-
- case PGSTAT_MTYPE_RESETSLRUCOUNTER:
- pgstat_recv_resetslrucounter(&msg.msg_resetslrucounter,
- len);
- break;
-
- case PGSTAT_MTYPE_RESETREPLSLOTCOUNTER:
- pgstat_recv_resetreplslotcounter(&msg.msg_resetreplslotcounter,
- len);
- break;
-
- case PGSTAT_MTYPE_AUTOVAC_START:
- pgstat_recv_autovac(&msg.msg_autovacuum_start, len);
- break;
-
- case PGSTAT_MTYPE_VACUUM:
- pgstat_recv_vacuum(&msg.msg_vacuum, len);
- break;
-
- case PGSTAT_MTYPE_ANALYZE:
- pgstat_recv_analyze(&msg.msg_analyze, len);
- break;
-
- case PGSTAT_MTYPE_ARCHIVER:
- pgstat_recv_archiver(&msg.msg_archiver, len);
- break;
-
- case PGSTAT_MTYPE_BGWRITER:
- pgstat_recv_bgwriter(&msg.msg_bgwriter, len);
- break;
-
- case PGSTAT_MTYPE_WAL:
- pgstat_recv_wal(&msg.msg_wal, len);
- break;
-
- case PGSTAT_MTYPE_SLRU:
- pgstat_recv_slru(&msg.msg_slru, len);
- break;
-
- case PGSTAT_MTYPE_FUNCSTAT:
- pgstat_recv_funcstat(&msg.msg_funcstat, len);
- break;
-
- case PGSTAT_MTYPE_FUNCPURGE:
- pgstat_recv_funcpurge(&msg.msg_funcpurge, len);
- break;
-
- case PGSTAT_MTYPE_RECOVERYCONFLICT:
- pgstat_recv_recoveryconflict(&msg.msg_recoveryconflict,
- len);
- break;
-
- case PGSTAT_MTYPE_DEADLOCK:
- pgstat_recv_deadlock(&msg.msg_deadlock, len);
- break;
-
- case PGSTAT_MTYPE_TEMPFILE:
- pgstat_recv_tempfile(&msg.msg_tempfile, len);
- break;
-
- case PGSTAT_MTYPE_CHECKSUMFAILURE:
- pgstat_recv_checksum_failure(&msg.msg_checksumfailure,
- len);
- break;
-
- case PGSTAT_MTYPE_REPLSLOT:
- pgstat_recv_replslot(&msg.msg_replslot, len);
- break;
-
- case PGSTAT_MTYPE_CONNECTION:
- pgstat_recv_connstat(&msg.msg_conn, len);
- break;
-
- default:
- break;
- }
- } /* end of inner message-processing loop */
-
- /* Sleep until there's something to do */
-#ifndef WIN32
- wr = WaitEventSetWait(wes, -1L, &event, 1, WAIT_EVENT_PGSTAT_MAIN);
-#else
-
- /*
- * Windows, at least in its Windows Server 2003 R2 incarnation,
- * sometimes loses FD_READ events. Waking up and retrying the recv()
- * fixes that, so don't sleep indefinitely. This is a crock of the
- * first water, but until somebody wants to debug exactly what's
- * happening there, this is the best we can do. The two-second
- * timeout matches our pre-9.2 behavior, and needs to be short enough
- * to not provoke "using stale statistics" complaints from
- * backend_read_statsfile.
- */
- wr = WaitEventSetWait(wes, 2 * 1000L /* msec */ , &event, 1,
- WAIT_EVENT_PGSTAT_MAIN);
-#endif
-
- /*
- * Emergency bailout if postmaster has died. This is to avoid the
- * necessity for manual cleanup of all postmaster children.
- */
- if (wr == 1 && event.events == WL_POSTMASTER_DEATH)
+ case DISCONNECT_CLIENT_EOF:
+ ldbstats->counts.n_sessions_abandoned++;
break;
- } /* end of outer loop */
-
- /*
- * Save the final stats to reuse at next startup.
- */
- pgstat_write_statsfiles(true, true);
-
- FreeWaitEventSet(wes);
-
- exit(0);
+ case DISCONNECT_FATAL:
+ ldbstats->counts.n_sessions_fatal++;
+ break;
+ case DISCONNECT_KILLED:
+ ldbstats->counts.n_sessions_killed++;
+ break;
+ }
}
-/*
- * Subroutine to clear stats in a database entry
+/* ----------
+ * get_local_dbstat_entry() -
*
- * Tables and functions hashes are initialized to empty.
- */
-static void
-reset_dbentry_counters(PgStat_StatDBEntry *dbentry)
-{
- HASHCTL hash_ctl;
-
- dbentry->n_xact_commit = 0;
- dbentry->n_xact_rollback = 0;
- dbentry->n_blocks_fetched = 0;
- dbentry->n_blocks_hit = 0;
- dbentry->n_tuples_returned = 0;
- dbentry->n_tuples_fetched = 0;
- dbentry->n_tuples_inserted = 0;
- dbentry->n_tuples_updated = 0;
- dbentry->n_tuples_deleted = 0;
- dbentry->last_autovac_time = 0;
- dbentry->n_conflict_tablespace = 0;
- dbentry->n_conflict_lock = 0;
- dbentry->n_conflict_snapshot = 0;
- dbentry->n_conflict_bufferpin = 0;
- dbentry->n_conflict_startup_deadlock = 0;
- dbentry->n_temp_files = 0;
- dbentry->n_temp_bytes = 0;
- dbentry->n_deadlocks = 0;
- dbentry->n_checksum_failures = 0;
- dbentry->last_checksum_failure = 0;
- dbentry->n_block_read_time = 0;
- dbentry->n_block_write_time = 0;
- dbentry->n_sessions = 0;
- dbentry->total_session_time = 0;
- dbentry->total_active_time = 0;
- dbentry->total_idle_in_xact_time = 0;
- dbentry->n_sessions_abandoned = 0;
- dbentry->n_sessions_fatal = 0;
- dbentry->n_sessions_killed = 0;
-
- dbentry->stat_reset_timestamp = GetCurrentTimestamp();
- dbentry->stats_timestamp = 0;
-
- hash_ctl.keysize = sizeof(Oid);
- hash_ctl.entrysize = sizeof(PgStat_StatTabEntry);
- dbentry->tables = hash_create("Per-database table",
- PGSTAT_TAB_HASH_SIZE,
- &hash_ctl,
- HASH_ELEM | HASH_BLOBS);
-
- hash_ctl.keysize = sizeof(Oid);
- hash_ctl.entrysize = sizeof(PgStat_StatFuncEntry);
- dbentry->functions = hash_create("Per-database function",
- PGSTAT_FUNCTION_HASH_SIZE,
- &hash_ctl,
- HASH_ELEM | HASH_BLOBS);
-}
-
-/*
- * Lookup the hash table entry for the specified database. If no hash
- * table entry exists, initialize it, if the create parameter is true.
- * Else, return NULL.
+ * Find or create a local PgStat_StatDBEntry entry for dbid. New entry is
+ * created and initialized if not exists.
*/
static PgStat_StatDBEntry *
-pgstat_get_db_entry(Oid databaseid, bool create)
+get_local_dbstat_entry(Oid dbid)
{
- PgStat_StatDBEntry *result;
+ PgStat_StatDBEntry *dbentry;
bool found;
- HASHACTION action = (create ? HASH_ENTER : HASH_FIND);
-
- /* Lookup or create the hash table entry for this database */
- result = (PgStat_StatDBEntry *) hash_search(pgStatDBHash,
- &databaseid,
- action, &found);
-
- if (!create && !found)
- return NULL;
/*
- * If not found, initialize the new one. This creates empty hash tables
- * for tables and functions, too.
+ * Find an entry or create a new one.
*/
- if (!found)
- reset_dbentry_counters(result);
+ dbentry = (PgStat_StatDBEntry *)
+ get_local_stat_entry(PGSTAT_TYPE_DB, dbid, InvalidOid,
+ true, &found);
- return result;
+ return dbentry;
}
-
-/*
- * Lookup the hash table entry for the specified table. If no hash
- * table entry exists, initialize it, if the create parameter is true.
- * Else, return NULL.
+/* ----------
+ * get_local_tabstat_entry() -
+ * Find or create a PgStat_TableStatus entry for rel. New entry is created and
+ * initialized if not exists.
+ * ----------
*/
-static PgStat_StatTabEntry *
-pgstat_get_tab_entry(PgStat_StatDBEntry *dbentry, Oid tableoid, bool create)
+static PgStat_TableStatus *
+get_local_tabstat_entry(Oid rel_id, bool isshared)
{
- PgStat_StatTabEntry *result;
+ PgStat_TableStatus *tabentry;
bool found;
- HASHACTION action = (create ? HASH_ENTER : HASH_FIND);
- /* Lookup or create the hash table entry for this table */
- result = (PgStat_StatTabEntry *) hash_search(dbentry->tables,
- &tableoid,
- action, &found);
+ tabentry = (PgStat_TableStatus *)
+ get_local_stat_entry(PGSTAT_TYPE_TABLE,
+ isshared ? InvalidOid : MyDatabaseId,
+ rel_id, true, &found);
- if (!create && !found)
- return NULL;
-
- /* If not found, initialize the new one. */
- if (!found)
- {
- result->numscans = 0;
- result->tuples_returned = 0;
- result->tuples_fetched = 0;
- result->tuples_inserted = 0;
- result->tuples_updated = 0;
- result->tuples_deleted = 0;
- result->tuples_hot_updated = 0;
- result->n_live_tuples = 0;
- result->n_dead_tuples = 0;
- result->changes_since_analyze = 0;
- result->inserts_since_vacuum = 0;
- result->blocks_fetched = 0;
- result->blocks_hit = 0;
- result->vacuum_timestamp = 0;
- result->vacuum_count = 0;
- result->autovac_vacuum_timestamp = 0;
- result->autovac_vacuum_count = 0;
- result->analyze_timestamp = 0;
- result->analyze_count = 0;
- result->autovac_analyze_timestamp = 0;
- result->autovac_analyze_count = 0;
- }
-
- return result;
+ return tabentry;
}
-
/* ----------
- * pgstat_write_statsfiles() -
- * Write the global statistics file, as well as requested DB files.
+ * pgstat_write_statsfile() -
+ * Write the global statistics file, as well as DB files.
*
- * 'permanent' specifies writing to the permanent files not temporary ones.
- * When true (happens only when the collector is shutting down), also remove
- * the temporary files so that backends starting up under a new postmaster
- * can't read old data before the new collector is ready.
- *
- * When 'allDbs' is false, only the requested databases (listed in
- * pending_write_requests) will be written; otherwise, all databases
- * will be written.
+ * This function is called in the last process that is accessing the shared
+ * stats so locking is not required.
* ----------
*/
static void
-pgstat_write_statsfiles(bool permanent, bool allDbs)
+pgstat_write_statsfile(void)
{
- HASH_SEQ_STATUS hstat;
- PgStat_StatDBEntry *dbentry;
FILE *fpout;
int32 format_id;
- const char *tmpfile = permanent ? PGSTAT_STAT_PERMANENT_TMPFILE : pgstat_stat_tmpname;
- const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename;
+ const char *tmpfile = PGSTAT_STAT_PERMANENT_TMPFILE;
+ const char *statfile = PGSTAT_STAT_PERMANENT_FILENAME;
int rc;
- int i;
+ dshash_seq_status hstat;
+ PgStatHashEntry *ps;
+
+ /* stats is not initialized yet. just return. */
+ if (StatsShmem->stats_dsa_handle == DSM_HANDLE_INVALID)
+ return;
+
+ /* this is the last process that is accesing the shared stats */
+#ifdef USE_ASSERT_CHECKING
+ LWLockAcquire(StatsLock, LW_SHARED);
+ Assert(StatsShmem->refcount == 0);
+ LWLockRelease(StatsLock);
+#endif
elog(DEBUG2, "writing stats file \"%s\"", statfile);
@@ -5267,7 +5661,7 @@ pgstat_write_statsfiles(bool permanent, bool allDbs)
/*
* Set the timestamp of the stats file.
*/
- globalStats.stats_timestamp = GetCurrentTimestamp();
+ pg_atomic_write_u64(&StatsShmem->stats_timestamp, GetCurrentTimestamp());
/*
* Write the file header --- currently just a format ID.
@@ -5277,200 +5671,72 @@ pgstat_write_statsfiles(bool permanent, bool allDbs)
(void) rc; /* we'll check for error with ferror */
/*
- * Write global stats struct
+ * Write bgwriter global stats struct
*/
- rc = fwrite(&globalStats, sizeof(globalStats), 1, fpout);
+ rc = fwrite(&StatsShmem->bgwriter_stats, sizeof(PgStat_BgWriter), 1, fpout);
(void) rc; /* we'll check for error with ferror */
/*
- * Write archiver stats struct
+ * Write checkpointer global stats struct
*/
- rc = fwrite(&archiverStats, sizeof(archiverStats), 1, fpout);
+ rc = fwrite(&StatsShmem->checkpointer_stats, sizeof(PgStat_CheckPointer), 1, fpout);
(void) rc; /* we'll check for error with ferror */
/*
- * Write WAL stats struct
+ * Write archiver global stats struct
*/
- rc = fwrite(&walStats, sizeof(walStats), 1, fpout);
+ rc = fwrite(&StatsShmem->archiver_stats, sizeof(PgStat_Archiver), 1,
+ fpout);
+ (void) rc; /* we'll check for error with ferror */
+
+ /*
+ * Write WAL global stats struct
+ */
+ rc = fwrite(&StatsShmem->wal_stats, sizeof(PgStat_Wal), 1, fpout);
(void) rc; /* we'll check for error with ferror */
/*
* Write SLRU stats struct
*/
- rc = fwrite(slruStats, sizeof(slruStats), 1, fpout);
+ rc = fwrite(&StatsShmem->slru_stats, sizeof(PgStatSharedSLRUStats), 1,
+ fpout);
(void) rc; /* we'll check for error with ferror */
/*
- * Walk through the database table.
+ * Walk through the stats entry
*/
- hash_seq_init(&hstat, pgStatDBHash);
- while ((dbentry = (PgStat_StatDBEntry *) hash_seq_search(&hstat)) != NULL)
+ dshash_seq_init(&hstat, pgStatSharedHash, false);
+ while ((ps = dshash_seq_next(&hstat)) != NULL)
{
- /*
- * Write out the table and function stats for this DB into the
- * appropriate per-DB stat file, if required.
- */
- if (allDbs || pgstat_db_requested(dbentry->databaseid))
+ PgStat_StatEntryHeader *shent;
+ size_t len;
+
+ CHECK_FOR_INTERRUPTS();
+
+ shent = (PgStat_StatEntryHeader *) dsa_get_address(area, ps->body);
+
+ /* we may have some "dropped" entries not yet removed, skip them */
+ if (shent->dropped)
+ continue;
+
+ /* Make DB's timestamp consistent with the global stats */
+ if (ps->key.type == PGSTAT_TYPE_DB)
{
- /* Make DB's timestamp consistent with the global stats */
- dbentry->stats_timestamp = globalStats.stats_timestamp;
+ PgStat_StatDBEntry *dbentry = (PgStat_StatDBEntry *) shent;
- pgstat_write_db_statsfile(dbentry, permanent);
+ dbentry->stats_timestamp =
+ (TimestampTz) pg_atomic_read_u64(&StatsShmem->stats_timestamp);
}
- /*
- * Write out the DB entry. We don't write the tables or functions
- * pointers, since they're of no use to any other process.
- */
- fputc('D', fpout);
- rc = fwrite(dbentry, offsetof(PgStat_StatDBEntry, tables), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
- }
-
- /*
- * Write replication slot stats struct
- */
- for (i = 0; i < nReplSlotStats; i++)
- {
- fputc('R', fpout);
- rc = fwrite(&replSlotStats[i], sizeof(PgStat_ReplSlotStats), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
- }
-
- /*
- * No more output to be done. Close the temp file and replace the old
- * pgstat.stat with it. The ferror() check replaces testing for error
- * after each individual fputc or fwrite above.
- */
- fputc('E', fpout);
-
- if (ferror(fpout))
- {
- ereport(LOG,
- (errcode_for_file_access(),
- errmsg("could not write temporary statistics file \"%s\": %m",
- tmpfile)));
- FreeFile(fpout);
- unlink(tmpfile);
- }
- else if (FreeFile(fpout) < 0)
- {
- ereport(LOG,
- (errcode_for_file_access(),
- errmsg("could not close temporary statistics file \"%s\": %m",
- tmpfile)));
- unlink(tmpfile);
- }
- else if (rename(tmpfile, statfile) < 0)
- {
- ereport(LOG,
- (errcode_for_file_access(),
- errmsg("could not rename temporary statistics file \"%s\" to \"%s\": %m",
- tmpfile, statfile)));
- unlink(tmpfile);
- }
-
- if (permanent)
- unlink(pgstat_stat_filename);
-
- /*
- * Now throw away the list of requests. Note that requests sent after we
- * started the write are still waiting on the network socket.
- */
- list_free(pending_write_requests);
- pending_write_requests = NIL;
-}
-
-/*
- * return the filename for a DB stat file; filename is the output buffer,
- * of length len.
- */
-static void
-get_dbstat_filename(bool permanent, bool tempname, Oid databaseid,
- char *filename, int len)
-{
- int printed;
-
- /* NB -- pgstat_reset_remove_files knows about the pattern this uses */
- printed = snprintf(filename, len, "%s/db_%u.%s",
- permanent ? PGSTAT_STAT_PERMANENT_DIRECTORY :
- pgstat_stat_directory,
- databaseid,
- tempname ? "tmp" : "stat");
- if (printed >= len)
- elog(ERROR, "overlength pgstat path");
-}
-
-/* ----------
- * pgstat_write_db_statsfile() -
- * Write the stat file for a single database.
- *
- * If writing to the permanent file (happens when the collector is
- * shutting down only), remove the temporary file so that backends
- * starting up under a new postmaster can't read the old data before
- * the new collector is ready.
- * ----------
- */
-static void
-pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent)
-{
- HASH_SEQ_STATUS tstat;
- HASH_SEQ_STATUS fstat;
- PgStat_StatTabEntry *tabentry;
- PgStat_StatFuncEntry *funcentry;
- FILE *fpout;
- int32 format_id;
- Oid dbid = dbentry->databaseid;
- int rc;
- char tmpfile[MAXPGPATH];
- char statfile[MAXPGPATH];
-
- get_dbstat_filename(permanent, true, dbid, tmpfile, MAXPGPATH);
- get_dbstat_filename(permanent, false, dbid, statfile, MAXPGPATH);
-
- elog(DEBUG2, "writing stats file \"%s\"", statfile);
-
- /*
- * Open the statistics temp file to write out the current values.
- */
- fpout = AllocateFile(tmpfile, PG_BINARY_W);
- if (fpout == NULL)
- {
- ereport(LOG,
- (errcode_for_file_access(),
- errmsg("could not open temporary statistics file \"%s\": %m",
- tmpfile)));
- return;
- }
-
- /*
- * Write the file header --- currently just a format ID.
- */
- format_id = PGSTAT_FILE_FORMAT_ID;
- rc = fwrite(&format_id, sizeof(format_id), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
-
- /*
- * Walk through the database's access stats per table.
- */
- hash_seq_init(&tstat, dbentry->tables);
- while ((tabentry = (PgStat_StatTabEntry *) hash_seq_search(&tstat)) != NULL)
- {
- fputc('T', fpout);
- rc = fwrite(tabentry, sizeof(PgStat_StatTabEntry), 1, fpout);
- (void) rc; /* we'll check for error with ferror */
- }
+ fputc('S', fpout);
+ rc = fwrite(&ps->key, sizeof(PgStatHashKey), 1, fpout);
- /*
- * Walk through the database's function stats table.
- */
- hash_seq_init(&fstat, dbentry->functions);
- while ((funcentry = (PgStat_StatFuncEntry *) hash_seq_search(&fstat)) != NULL)
- {
- fputc('F', fpout);
- rc = fwrite(funcentry, sizeof(PgStat_StatFuncEntry), 1, fpout);
+ /* Write except the header part of the etnry */
+ len = PGSTAT_SHENT_BODY_LEN(ps->key.type);
+ rc = fwrite(PGSTAT_SHENT_BODY(shent), len, 1, fpout);
(void) rc; /* we'll check for error with ferror */
}
+ dshash_seq_term(&hstat);
/*
* No more output to be done. Close the temp file and replace the old
@@ -5504,113 +5770,65 @@ pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent)
tmpfile, statfile)));
unlink(tmpfile);
}
-
- if (permanent)
- {
- get_dbstat_filename(false, false, dbid, statfile, MAXPGPATH);
-
- elog(DEBUG2, "removing temporary stats file \"%s\"", statfile);
- unlink(statfile);
- }
}
/* ----------
- * pgstat_read_statsfiles() -
- *
- * Reads in some existing statistics collector files and returns the
- * databases hash table that is the top level of the data.
+ * pgstat_read_statsfile() -
*
- * If 'onlydb' is not InvalidOid, it means we only want data for that DB
- * plus the shared catalogs ("DB 0"). We'll still populate the DB hash
- * table for all databases, but we don't bother even creating table/function
- * hash tables for other databases.
+ * Reads in existing activity statistics file into the shared stats hash.
*
- * 'permanent' specifies reading from the permanent files not temporary ones.
- * When true (happens only when the collector is starting up), remove the
- * files after reading; the in-memory status is now authoritative, and the
- * files would be out of date in case somebody else reads them.
- *
- * If a 'deep' read is requested, table/function stats are read, otherwise
- * the table/function hash tables remain empty.
+ * This function is called in the only process that is accessing the shared
+ * stats so locking is not required.
* ----------
*/
-static HTAB *
-pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
+static void
+pgstat_read_statsfile(void)
{
- PgStat_StatDBEntry *dbentry;
- PgStat_StatDBEntry dbbuf;
- HASHCTL hash_ctl;
- HTAB *dbhash;
FILE *fpin;
int32 format_id;
bool found;
- const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename;
- int i;
+ const char *statfile = PGSTAT_STAT_PERMANENT_FILENAME;
+ char tag;
- /*
- * The tables will live in pgStatLocalContext.
- */
- pgstat_setup_memcxt();
+ /* shouldn't be called from postmaster */
+ Assert(IsUnderPostmaster);
- /*
- * Create the DB hashtable
- */
- hash_ctl.keysize = sizeof(Oid);
- hash_ctl.entrysize = sizeof(PgStat_StatDBEntry);
- hash_ctl.hcxt = pgStatLocalContext;
- dbhash = hash_create("Databases hash", PGSTAT_DB_HASH_SIZE, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ /* this is the only process that is accesing the shared stats */
+#ifdef USE_ASSERT_CHECKING
+ LWLockAcquire(StatsLock, LW_SHARED);
+ Assert(StatsShmem->refcount == 1);
+ LWLockRelease(StatsLock);
+#endif
- /* Allocate the space for replication slot statistics */
- replSlotStats = palloc0(max_replication_slots * sizeof(PgStat_ReplSlotStats));
- nReplSlotStats = 0;
-
- /*
- * Clear out global, archiver, WAL and SLRU statistics so they start from
- * zero in case we can't load an existing statsfile.
- */
- memset(&globalStats, 0, sizeof(globalStats));
- memset(&archiverStats, 0, sizeof(archiverStats));
- memset(&walStats, 0, sizeof(walStats));
- memset(&slruStats, 0, sizeof(slruStats));
+ elog(DEBUG2, "reading stats file \"%s\"", statfile);
/*
* Set the current timestamp (will be kept only in case we can't load an
* existing statsfile).
*/
- globalStats.stat_reset_timestamp = GetCurrentTimestamp();
- archiverStats.stat_reset_timestamp = globalStats.stat_reset_timestamp;
- walStats.stat_reset_timestamp = globalStats.stat_reset_timestamp;
-
- /*
- * Set the same reset timestamp for all SLRU items too.
- */
- for (i = 0; i < SLRU_NUM_ELEMENTS; i++)
- slruStats[i].stat_reset_timestamp = globalStats.stat_reset_timestamp;
-
- /*
- * Set the same reset timestamp for all replication slots too.
- */
- for (i = 0; i < max_replication_slots; i++)
- replSlotStats[i].stat_reset_timestamp = globalStats.stat_reset_timestamp;
+ StatsShmem->bgwriter_stats.stat_reset_timestamp = GetCurrentTimestamp();
+ StatsShmem->archiver_stats.stat_reset_timestamp =
+ StatsShmem->bgwriter_stats.stat_reset_timestamp;
+ StatsShmem->wal_stats.stat_reset_timestamp =
+ StatsShmem->bgwriter_stats.stat_reset_timestamp;
/*
* Try to open the stats file. If it doesn't exist, the backends simply
- * return zero for anything and the collector simply starts from scratch
- * with empty counters.
+ * returns zero for anything and the activity statistics simply starts
+ * from scratch with empty counters.
*
- * ENOENT is a possibility if the stats collector is not running or has
- * not yet written the stats file the first time. Any other failure
+ * ENOENT is a possibility if the activity statistics is not running or
+ * has not yet written the stats file the first time. Any other failure
* condition is suspicious.
*/
if ((fpin = AllocateFile(statfile, PG_BINARY_R)) == NULL)
{
if (errno != ENOENT)
- ereport(pgStatRunningInCollector ? LOG : WARNING,
+ ereport(LOG,
(errcode_for_file_access(),
errmsg("could not open statistics file \"%s\": %m",
statfile)));
- return dbhash;
+ return;
}
/*
@@ -5619,681 +5837,150 @@ pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep)
if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
format_id != PGSTAT_FILE_FORMAT_ID)
{
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"", statfile)));
- goto done;
- }
-
- /*
- * Read global stats struct
- */
- if (fread(&globalStats, 1, sizeof(globalStats), fpin) != sizeof(globalStats))
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"", statfile)));
- memset(&globalStats, 0, sizeof(globalStats));
- goto done;
- }
-
- /*
- * In the collector, disregard the timestamp we read from the permanent
- * stats file; we should be willing to write a temp stats file immediately
- * upon the first request from any backend. This only matters if the old
- * file's timestamp is less than PGSTAT_STAT_INTERVAL ago, but that's not
- * an unusual scenario.
- */
- if (pgStatRunningInCollector)
- globalStats.stats_timestamp = 0;
-
- /*
- * Read archiver stats struct
- */
- if (fread(&archiverStats, 1, sizeof(archiverStats), fpin) != sizeof(archiverStats))
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"", statfile)));
- memset(&archiverStats, 0, sizeof(archiverStats));
- goto done;
- }
-
- /*
- * Read WAL stats struct
- */
- if (fread(&walStats, 1, sizeof(walStats), fpin) != sizeof(walStats))
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
+ ereport(LOG,
(errmsg("corrupted statistics file \"%s\"", statfile)));
- memset(&walStats, 0, sizeof(walStats));
goto done;
}
/*
- * Read SLRU stats struct
+ * Read bgwiter stats struct
*/
- if (fread(slruStats, 1, sizeof(slruStats), fpin) != sizeof(slruStats))
+ if (fread(&StatsShmem->bgwriter_stats, 1, sizeof(PgStat_BgWriter), fpin) !=
+ sizeof(PgStat_BgWriter))
{
- ereport(pgStatRunningInCollector ? LOG : WARNING,
+ ereport(LOG,
(errmsg("corrupted statistics file \"%s\"", statfile)));
- memset(&slruStats, 0, sizeof(slruStats));
+ MemSet(&StatsShmem->bgwriter_stats, 0, sizeof(PgStat_BgWriter));
goto done;
}
/*
- * We found an existing collector stats file. Read it and put all the
- * hashtable entries into place.
- */
- for (;;)
- {
- switch (fgetc(fpin))
- {
- /*
- * 'D' A PgStat_StatDBEntry struct describing a database
- * follows.
- */
- case 'D':
- if (fread(&dbbuf, 1, offsetof(PgStat_StatDBEntry, tables),
- fpin) != offsetof(PgStat_StatDBEntry, tables))
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"",
- statfile)));
- goto done;
- }
-
- /*
- * Add to the DB hash
- */
- dbentry = (PgStat_StatDBEntry *) hash_search(dbhash,
- (void *) &dbbuf.databaseid,
- HASH_ENTER,
- &found);
- if (found)
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"",
- statfile)));
- goto done;
- }
-
- memcpy(dbentry, &dbbuf, sizeof(PgStat_StatDBEntry));
- dbentry->tables = NULL;
- dbentry->functions = NULL;
-
- /*
- * In the collector, disregard the timestamp we read from the
- * permanent stats file; we should be willing to write a temp
- * stats file immediately upon the first request from any
- * backend.
- */
- if (pgStatRunningInCollector)
- dbentry->stats_timestamp = 0;
-
- /*
- * Don't create tables/functions hashtables for uninteresting
- * databases.
- */
- if (onlydb != InvalidOid)
- {
- if (dbbuf.databaseid != onlydb &&
- dbbuf.databaseid != InvalidOid)
- break;
- }
-
- hash_ctl.keysize = sizeof(Oid);
- hash_ctl.entrysize = sizeof(PgStat_StatTabEntry);
- hash_ctl.hcxt = pgStatLocalContext;
- dbentry->tables = hash_create("Per-database table",
- PGSTAT_TAB_HASH_SIZE,
- &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
-
- hash_ctl.keysize = sizeof(Oid);
- hash_ctl.entrysize = sizeof(PgStat_StatFuncEntry);
- hash_ctl.hcxt = pgStatLocalContext;
- dbentry->functions = hash_create("Per-database function",
- PGSTAT_FUNCTION_HASH_SIZE,
- &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
-
- /*
- * If requested, read the data from the database-specific
- * file. Otherwise we just leave the hashtables empty.
- */
- if (deep)
- pgstat_read_db_statsfile(dbentry->databaseid,
- dbentry->tables,
- dbentry->functions,
- permanent);
-
- break;
-
- /*
- * 'R' A PgStat_ReplSlotStats struct describing a replication
- * slot follows.
- */
- case 'R':
- if (fread(&replSlotStats[nReplSlotStats], 1, sizeof(PgStat_ReplSlotStats), fpin)
- != sizeof(PgStat_ReplSlotStats))
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"",
- statfile)));
- memset(&replSlotStats[nReplSlotStats], 0, sizeof(PgStat_ReplSlotStats));
- goto done;
- }
- nReplSlotStats++;
- break;
-
- case 'E':
- goto done;
-
- default:
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"",
- statfile)));
- goto done;
- }
- }
-
-done:
- FreeFile(fpin);
-
- /* If requested to read the permanent file, also get rid of it. */
- if (permanent)
- {
- elog(DEBUG2, "removing permanent stats file \"%s\"", statfile);
- unlink(statfile);
- }
-
- return dbhash;
-}
-
-
-/* ----------
- * pgstat_read_db_statsfile() -
- *
- * Reads in the existing statistics collector file for the given database,
- * filling the passed-in tables and functions hash tables.
- *
- * As in pgstat_read_statsfiles, if the permanent file is requested, it is
- * removed after reading.
- *
- * Note: this code has the ability to skip storing per-table or per-function
- * data, if NULL is passed for the corresponding hashtable. That's not used
- * at the moment though.
- * ----------
- */
-static void
-pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash,
- bool permanent)
-{
- PgStat_StatTabEntry *tabentry;
- PgStat_StatTabEntry tabbuf;
- PgStat_StatFuncEntry funcbuf;
- PgStat_StatFuncEntry *funcentry;
- FILE *fpin;
- int32 format_id;
- bool found;
- char statfile[MAXPGPATH];
-
- get_dbstat_filename(permanent, false, databaseid, statfile, MAXPGPATH);
-
- /*
- * Try to open the stats file. If it doesn't exist, the backends simply
- * return zero for anything and the collector simply starts from scratch
- * with empty counters.
- *
- * ENOENT is a possibility if the stats collector is not running or has
- * not yet written the stats file the first time. Any other failure
- * condition is suspicious.
- */
- if ((fpin = AllocateFile(statfile, PG_BINARY_R)) == NULL)
- {
- if (errno != ENOENT)
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errcode_for_file_access(),
- errmsg("could not open statistics file \"%s\": %m",
- statfile)));
- return;
- }
-
- /*
- * Verify it's of the expected format.
+ * Read checkpointer stats struct
*/
- if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
- format_id != PGSTAT_FILE_FORMAT_ID)
+ if (fread(&StatsShmem->checkpointer_stats, 1, sizeof(PgStat_CheckPointer), fpin) !=
+ sizeof(PgStat_CheckPointer))
{
- ereport(pgStatRunningInCollector ? LOG : WARNING,
+ ereport(LOG,
(errmsg("corrupted statistics file \"%s\"", statfile)));
+ MemSet(&StatsShmem->checkpointer_stats, 0, sizeof(PgStat_CheckPointer));
goto done;
}
- /*
- * We found an existing collector stats file. Read it and put all the
- * hashtable entries into place.
- */
- for (;;)
- {
- switch (fgetc(fpin))
- {
- /*
- * 'T' A PgStat_StatTabEntry follows.
- */
- case 'T':
- if (fread(&tabbuf, 1, sizeof(PgStat_StatTabEntry),
- fpin) != sizeof(PgStat_StatTabEntry))
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"",
- statfile)));
- goto done;
- }
-
- /*
- * Skip if table data not wanted.
- */
- if (tabhash == NULL)
- break;
-
- tabentry = (PgStat_StatTabEntry *) hash_search(tabhash,
- (void *) &tabbuf.tableid,
- HASH_ENTER, &found);
-
- if (found)
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"",
- statfile)));
- goto done;
- }
-
- memcpy(tabentry, &tabbuf, sizeof(tabbuf));
- break;
-
- /*
- * 'F' A PgStat_StatFuncEntry follows.
- */
- case 'F':
- if (fread(&funcbuf, 1, sizeof(PgStat_StatFuncEntry),
- fpin) != sizeof(PgStat_StatFuncEntry))
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"",
- statfile)));
- goto done;
- }
-
- /*
- * Skip if function data not wanted.
- */
- if (funchash == NULL)
- break;
-
- funcentry = (PgStat_StatFuncEntry *) hash_search(funchash,
- (void *) &funcbuf.functionid,
- HASH_ENTER, &found);
-
- if (found)
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"",
- statfile)));
- goto done;
- }
-
- memcpy(funcentry, &funcbuf, sizeof(funcbuf));
- break;
-
- /*
- * 'E' The EOF marker of a complete stats file.
- */
- case 'E':
- goto done;
-
- default:
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"",
- statfile)));
- goto done;
- }
- }
-
-done:
- FreeFile(fpin);
-
- if (permanent)
- {
- elog(DEBUG2, "removing permanent stats file \"%s\"", statfile);
- unlink(statfile);
- }
-}
-
-/* ----------
- * pgstat_read_db_statsfile_timestamp() -
- *
- * Attempt to determine the timestamp of the last db statfile write.
- * Returns true if successful; the timestamp is stored in *ts. The caller must
- * rely on timestamp stored in *ts iff the function returns true.
- *
- * This needs to be careful about handling databases for which no stats file
- * exists, such as databases without a stat entry or those not yet written:
- *
- * - if there's a database entry in the global file, return the corresponding
- * stats_timestamp value.
- *
- * - if there's no db stat entry (e.g. for a new or inactive database),
- * there's no stats_timestamp value, but also nothing to write so we return
- * the timestamp of the global statfile.
- * ----------
- */
-static bool
-pgstat_read_db_statsfile_timestamp(Oid databaseid, bool permanent,
- TimestampTz *ts)
-{
- PgStat_StatDBEntry dbentry;
- PgStat_GlobalStats myGlobalStats;
- PgStat_ArchiverStats myArchiverStats;
- PgStat_WalStats myWalStats;
- PgStat_SLRUStats mySLRUStats[SLRU_NUM_ELEMENTS];
- PgStat_ReplSlotStats myReplSlotStats;
- FILE *fpin;
- int32 format_id;
- const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename;
-
- /*
- * Try to open the stats file. As above, anything but ENOENT is worthy of
- * complaining about.
- */
- if ((fpin = AllocateFile(statfile, PG_BINARY_R)) == NULL)
- {
- if (errno != ENOENT)
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errcode_for_file_access(),
- errmsg("could not open statistics file \"%s\": %m",
- statfile)));
- return false;
- }
-
- /*
- * Verify it's of the expected format.
- */
- if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) ||
- format_id != PGSTAT_FILE_FORMAT_ID)
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"", statfile)));
- FreeFile(fpin);
- return false;
- }
-
- /*
- * Read global stats struct
- */
- if (fread(&myGlobalStats, 1, sizeof(myGlobalStats),
- fpin) != sizeof(myGlobalStats))
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"", statfile)));
- FreeFile(fpin);
- return false;
- }
-
/*
* Read archiver stats struct
*/
- if (fread(&myArchiverStats, 1, sizeof(myArchiverStats),
- fpin) != sizeof(myArchiverStats))
+ if (fread(&StatsShmem->archiver_stats, 1, sizeof(PgStat_Archiver),
+ fpin) != sizeof(PgStat_Archiver))
{
- ereport(pgStatRunningInCollector ? LOG : WARNING,
+ ereport(LOG,
(errmsg("corrupted statistics file \"%s\"", statfile)));
- FreeFile(fpin);
- return false;
+ MemSet(&StatsShmem->archiver_stats, 0, sizeof(PgStat_Archiver));
+ goto done;
}
/*
* Read WAL stats struct
*/
- if (fread(&myWalStats, 1, sizeof(myWalStats), fpin) != sizeof(myWalStats))
+ if (fread(&StatsShmem->wal_stats, 1, sizeof(PgStat_Wal), fpin)
+ != sizeof(PgStat_Wal))
{
- ereport(pgStatRunningInCollector ? LOG : WARNING,
+ ereport(LOG,
(errmsg("corrupted statistics file \"%s\"", statfile)));
- FreeFile(fpin);
- return false;
+ MemSet(&StatsShmem->wal_stats, 0, sizeof(PgStat_Wal));
+ goto done;
}
/*
* Read SLRU stats struct
*/
- if (fread(mySLRUStats, 1, sizeof(mySLRUStats), fpin) != sizeof(mySLRUStats))
+ if (fread(&StatsShmem->slru_stats, 1, sizeof(PgStatSharedSLRUStats),
+ fpin) != sizeof(PgStatSharedSLRUStats))
{
- ereport(pgStatRunningInCollector ? LOG : WARNING,
+ ereport(LOG,
(errmsg("corrupted statistics file \"%s\"", statfile)));
- FreeFile(fpin);
- return false;
- }
-
- /* By default, we're going to return the timestamp of the global file. */
- *ts = myGlobalStats.stats_timestamp;
-
- /*
- * We found an existing collector stats file. Read it and look for a
- * record for the requested database. If found, use its timestamp.
- */
- for (;;)
- {
- switch (fgetc(fpin))
- {
- /*
- * 'D' A PgStat_StatDBEntry struct describing a database
- * follows.
- */
- case 'D':
- if (fread(&dbentry, 1, offsetof(PgStat_StatDBEntry, tables),
- fpin) != offsetof(PgStat_StatDBEntry, tables))
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"",
- statfile)));
- FreeFile(fpin);
- return false;
- }
-
- /*
- * If this is the DB we're looking for, save its timestamp and
- * we're done.
- */
- if (dbentry.databaseid == databaseid)
- {
- *ts = dbentry.stats_timestamp;
- goto done;
- }
-
- break;
-
- /*
- * 'R' A PgStat_ReplSlotStats struct describing a replication
- * slot follows.
- */
- case 'R':
- if (fread(&myReplSlotStats, 1, sizeof(PgStat_ReplSlotStats), fpin)
- != sizeof(PgStat_ReplSlotStats))
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"",
- statfile)));
- FreeFile(fpin);
- return false;
- }
- break;
-
- case 'E':
- goto done;
-
- default:
- {
- ereport(pgStatRunningInCollector ? LOG : WARNING,
- (errmsg("corrupted statistics file \"%s\"",
- statfile)));
- FreeFile(fpin);
- return false;
- }
- }
+ goto done;
}
-done:
- FreeFile(fpin);
- return true;
-}
-
-/*
- * If not already done, read the statistics collector stats file into
- * some hash tables. The results will be kept until pgstat_clear_snapshot()
- * is called (typically, at end of transaction).
- */
-static void
-backend_read_statsfile(void)
-{
- TimestampTz min_ts = 0;
- TimestampTz ref_ts = 0;
- Oid inquiry_db;
- int count;
-
- /* already read it? */
- if (pgStatDBHash)
- return;
- Assert(!pgStatRunningInCollector);
-
- /*
- * In a normal backend, we check staleness of the data for our own DB, and
- * so we send MyDatabaseId in inquiry messages. In the autovac launcher,
- * check staleness of the shared-catalog data, and send InvalidOid in
- * inquiry messages so as not to force writing unnecessary data.
- */
- if (IsAutoVacuumLauncherProcess())
- inquiry_db = InvalidOid;
- else
- inquiry_db = MyDatabaseId;
-
/*
- * Loop until fresh enough stats file is available or we ran out of time.
- * The stats inquiry message is sent repeatedly in case collector drops
- * it; but not every single time, as that just swamps the collector.
+ * We found an existing activity statistics file. Read it and put all the
+ * hash table entries into place.
*/
- for (count = 0; count < PGSTAT_POLL_LOOP_COUNT; count++)
+ while ((tag = fgetc(fpin)) == 'S')
{
- bool ok;
- TimestampTz file_ts = 0;
- TimestampTz cur_ts;
+ PgStatHashKey key;
+ PgStat_StatEntryHeader *p;
+ size_t len;
CHECK_FOR_INTERRUPTS();
- ok = pgstat_read_db_statsfile_timestamp(inquiry_db, false, &file_ts);
-
- cur_ts = GetCurrentTimestamp();
- /* Calculate min acceptable timestamp, if we didn't already */
- if (count == 0 || cur_ts < ref_ts)
+ if (fread(&key, 1, sizeof(key), fpin) != sizeof(key))
{
- /*
- * We set the minimum acceptable timestamp to PGSTAT_STAT_INTERVAL
- * msec before now. This indirectly ensures that the collector
- * needn't write the file more often than PGSTAT_STAT_INTERVAL. In
- * an autovacuum worker, however, we want a lower delay to avoid
- * using stale data, so we use PGSTAT_RETRY_DELAY (since the
- * number of workers is low, this shouldn't be a problem).
- *
- * We don't recompute min_ts after sleeping, except in the
- * unlikely case that cur_ts went backwards. So we might end up
- * accepting a file a bit older than PGSTAT_STAT_INTERVAL. In
- * practice that shouldn't happen, though, as long as the sleep
- * time is less than PGSTAT_STAT_INTERVAL; and we don't want to
- * tell the collector that our cutoff time is less than what we'd
- * actually accept.
- */
- ref_ts = cur_ts;
- if (IsAutoVacuumWorkerProcess())
- min_ts = TimestampTzPlusMilliseconds(ref_ts,
- -PGSTAT_RETRY_DELAY);
- else
- min_ts = TimestampTzPlusMilliseconds(ref_ts,
- -PGSTAT_STAT_INTERVAL);
+ ereport(LOG,
+ (errmsg("corrupted statistics file \"%s\"", statfile)));
+ goto done;
}
- /*
- * If the file timestamp is actually newer than cur_ts, we must have
- * had a clock glitch (system time went backwards) or there is clock
- * skew between our processor and the stats collector's processor.
- * Accept the file, but send an inquiry message anyway to make
- * pgstat_recv_inquiry do a sanity check on the collector's time.
- */
- if (ok && file_ts > cur_ts)
- {
- /*
- * A small amount of clock skew between processors isn't terribly
- * surprising, but a large difference is worth logging. We
- * arbitrarily define "large" as 1000 msec.
- */
- if (file_ts >= TimestampTzPlusMilliseconds(cur_ts, 1000))
- {
- char *filetime;
- char *mytime;
-
- /* Copy because timestamptz_to_str returns a static buffer */
- filetime = pstrdup(timestamptz_to_str(file_ts));
- mytime = pstrdup(timestamptz_to_str(cur_ts));
- ereport(LOG,
- (errmsg("statistics collector's time %s is later than backend local time %s",
- filetime, mytime)));
- pfree(filetime);
- pfree(mytime);
- }
+ p = get_stat_entry(key.type, key.databaseid, key.objectid,
+ false, true, &found);
- pgstat_send_inquiry(cur_ts, min_ts, inquiry_db);
- break;
+ /* don't allow duplicate entries */
+ if (found)
+ {
+ ereport(LOG,
+ (errmsg("corrupted statistics file \"%s\"",
+ statfile)));
+ goto done;
}
- /* Normal acceptance case: file is not older than cutoff time */
- if (ok && file_ts >= min_ts)
- break;
+ /* Avoid overwriting header part */
+ len = PGSTAT_SHENT_BODY_LEN(key.type);
- /* Not there or too old, so kick the collector and wait a bit */
- if ((count % PGSTAT_INQ_LOOP_COUNT) == 0)
- pgstat_send_inquiry(cur_ts, min_ts, inquiry_db);
-
- pg_usleep(PGSTAT_RETRY_DELAY * 1000L);
+ if (fread(PGSTAT_SHENT_BODY(p), 1, len, fpin) != len)
+ {
+ ereport(LOG,
+ (errmsg("corrupted statistics file \"%s\"", statfile)));
+ goto done;
+ }
}
- if (count >= PGSTAT_POLL_LOOP_COUNT)
+ if (tag != 'E')
+ {
ereport(LOG,
- (errmsg("using stale statistics instead of current ones "
- "because stats collector is not responding")));
+ (errmsg("corrupted statistics file \"%s\"",
+ statfile)));
+ goto done;
+ }
- /*
- * Autovacuum launcher wants stats about all databases, but a shallow read
- * is sufficient. Regular backends want a deep read for just the tables
- * they can see (MyDatabaseId + shared catalogs).
- */
- if (IsAutoVacuumLauncherProcess())
- pgStatDBHash = pgstat_read_statsfiles(InvalidOid, false, false);
- else
- pgStatDBHash = pgstat_read_statsfiles(MyDatabaseId, false, true);
+done:
+ FreeFile(fpin);
+
+ elog(DEBUG2, "removing permanent stats file \"%s\"", statfile);
+ unlink(statfile);
+
+ return;
}
-
/* ----------
* pgstat_setup_memcxt() -
*
- * Create pgStatLocalContext, if not already done.
+ * Create pgStatLocalContext if not already done.
* ----------
*/
static void
pgstat_setup_memcxt(void)
{
if (!pgStatLocalContext)
- pgStatLocalContext = AllocSetContextCreate(TopMemoryContext,
- "Statistics snapshot",
- ALLOCSET_SMALL_SIZES);
-}
+ pgStatLocalContext =
+ AllocSetContextCreate(TopMemoryContext,
+ "Backend statistics snapshot",
+ ALLOCSET_SMALL_SIZES);
+ if (!pgStatCacheContext)
+ pgStatCacheContext =
+ AllocSetContextCreate(CacheMemoryContext,
+ "Activity statistics",
+ ALLOCSET_SMALL_SIZES);
+}
/* ----------
* pgstat_clear_snapshot() -
@@ -6310,945 +5997,25 @@ pgstat_clear_snapshot(void)
{
/* Release memory, if any was allocated */
if (pgStatLocalContext)
+ {
MemoryContextDelete(pgStatLocalContext);
- /* Reset variables */
- pgStatLocalContext = NULL;
- pgStatDBHash = NULL;
- localBackendStatusTable = NULL;
- localNumBackends = 0;
-}
-
-
-/* ----------
- * pgstat_recv_inquiry() -
- *
- * Process stat inquiry requests.
- * ----------
- */
-static void
-pgstat_recv_inquiry(PgStat_MsgInquiry *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
-
- elog(DEBUG2, "received inquiry for database %u", msg->databaseid);
-
- /*
- * If there's already a write request for this DB, there's nothing to do.
- *
- * Note that if a request is found, we return early and skip the below
- * check for clock skew. This is okay, since the only way for a DB
- * request to be present in the list is that we have been here since the
- * last write round. It seems sufficient to check for clock skew once per
- * write round.
- */
- if (list_member_oid(pending_write_requests, msg->databaseid))
- return;
-
- /*
- * Check to see if we last wrote this database at a time >= the requested
- * cutoff time. If so, this is a stale request that was generated before
- * we updated the DB file, and we don't need to do so again.
- *
- * If the requestor's local clock time is older than stats_timestamp, we
- * should suspect a clock glitch, ie system time going backwards; though
- * the more likely explanation is just delayed message receipt. It is
- * worth expending a GetCurrentTimestamp call to be sure, since a large
- * retreat in the system clock reading could otherwise cause us to neglect
- * to update the stats file for a long time.
- */
- dbentry = pgstat_get_db_entry(msg->databaseid, false);
- if (dbentry == NULL)
- {
- /*
- * We have no data for this DB. Enter a write request anyway so that
- * the global stats will get updated. This is needed to prevent
- * backend_read_statsfile from waiting for data that we cannot supply,
- * in the case of a new DB that nobody has yet reported any stats for.
- * See the behavior of pgstat_read_db_statsfile_timestamp.
- */
- }
- else if (msg->clock_time < dbentry->stats_timestamp)
- {
- TimestampTz cur_ts = GetCurrentTimestamp();
-
- if (cur_ts < dbentry->stats_timestamp)
- {
- /*
- * Sure enough, time went backwards. Force a new stats file write
- * to get back in sync; but first, log a complaint.
- */
- char *writetime;
- char *mytime;
-
- /* Copy because timestamptz_to_str returns a static buffer */
- writetime = pstrdup(timestamptz_to_str(dbentry->stats_timestamp));
- mytime = pstrdup(timestamptz_to_str(cur_ts));
- ereport(LOG,
- (errmsg("stats_timestamp %s is later than collector's time %s for database %u",
- writetime, mytime, dbentry->databaseid)));
- pfree(writetime);
- pfree(mytime);
- }
- else
- {
- /*
- * Nope, it's just an old request. Assuming msg's clock_time is
- * >= its cutoff_time, it must be stale, so we can ignore it.
- */
- return;
- }
- }
- else if (msg->cutoff_time <= dbentry->stats_timestamp)
- {
- /* Stale request, ignore it */
- return;
- }
-
- /*
- * We need to write this DB, so create a request.
- */
- pending_write_requests = lappend_oid(pending_write_requests,
- msg->databaseid);
-}
-
-
-/* ----------
- * pgstat_recv_tabstat() -
- *
- * Count what the backend has done.
- * ----------
- */
-static void
-pgstat_recv_tabstat(PgStat_MsgTabstat *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
- PgStat_StatTabEntry *tabentry;
- int i;
- bool found;
-
- dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
-
- /*
- * Update database-wide stats.
- */
- dbentry->n_xact_commit += (PgStat_Counter) (msg->m_xact_commit);
- dbentry->n_xact_rollback += (PgStat_Counter) (msg->m_xact_rollback);
- dbentry->n_block_read_time += msg->m_block_read_time;
- dbentry->n_block_write_time += msg->m_block_write_time;
-
- /*
- * Process all table entries in the message.
- */
- for (i = 0; i < msg->m_nentries; i++)
- {
- PgStat_TableEntry *tabmsg = &(msg->m_entry[i]);
-
- tabentry = (PgStat_StatTabEntry *) hash_search(dbentry->tables,
- (void *) &(tabmsg->t_id),
- HASH_ENTER, &found);
-
- if (!found)
- {
- /*
- * If it's a new table entry, initialize counters to the values we
- * just got.
- */
- tabentry->numscans = tabmsg->t_counts.t_numscans;
- tabentry->tuples_returned = tabmsg->t_counts.t_tuples_returned;
- tabentry->tuples_fetched = tabmsg->t_counts.t_tuples_fetched;
- tabentry->tuples_inserted = tabmsg->t_counts.t_tuples_inserted;
- tabentry->tuples_updated = tabmsg->t_counts.t_tuples_updated;
- tabentry->tuples_deleted = tabmsg->t_counts.t_tuples_deleted;
- tabentry->tuples_hot_updated = tabmsg->t_counts.t_tuples_hot_updated;
- tabentry->n_live_tuples = tabmsg->t_counts.t_delta_live_tuples;
- tabentry->n_dead_tuples = tabmsg->t_counts.t_delta_dead_tuples;
- tabentry->changes_since_analyze = tabmsg->t_counts.t_changed_tuples;
- tabentry->inserts_since_vacuum = tabmsg->t_counts.t_tuples_inserted;
- tabentry->blocks_fetched = tabmsg->t_counts.t_blocks_fetched;
- tabentry->blocks_hit = tabmsg->t_counts.t_blocks_hit;
-
- tabentry->vacuum_timestamp = 0;
- tabentry->vacuum_count = 0;
- tabentry->autovac_vacuum_timestamp = 0;
- tabentry->autovac_vacuum_count = 0;
- tabentry->analyze_timestamp = 0;
- tabentry->analyze_count = 0;
- tabentry->autovac_analyze_timestamp = 0;
- tabentry->autovac_analyze_count = 0;
- }
- else
- {
- /*
- * Otherwise add the values to the existing entry.
- */
- tabentry->numscans += tabmsg->t_counts.t_numscans;
- tabentry->tuples_returned += tabmsg->t_counts.t_tuples_returned;
- tabentry->tuples_fetched += tabmsg->t_counts.t_tuples_fetched;
- tabentry->tuples_inserted += tabmsg->t_counts.t_tuples_inserted;
- tabentry->tuples_updated += tabmsg->t_counts.t_tuples_updated;
- tabentry->tuples_deleted += tabmsg->t_counts.t_tuples_deleted;
- tabentry->tuples_hot_updated += tabmsg->t_counts.t_tuples_hot_updated;
- /* If table was truncated, first reset the live/dead counters */
- if (tabmsg->t_counts.t_truncated)
- {
- tabentry->n_live_tuples = 0;
- tabentry->n_dead_tuples = 0;
- tabentry->inserts_since_vacuum = 0;
- }
- tabentry->n_live_tuples += tabmsg->t_counts.t_delta_live_tuples;
- tabentry->n_dead_tuples += tabmsg->t_counts.t_delta_dead_tuples;
- tabentry->changes_since_analyze += tabmsg->t_counts.t_changed_tuples;
- tabentry->inserts_since_vacuum += tabmsg->t_counts.t_tuples_inserted;
- tabentry->blocks_fetched += tabmsg->t_counts.t_blocks_fetched;
- tabentry->blocks_hit += tabmsg->t_counts.t_blocks_hit;
- }
-
- /* Clamp n_live_tuples in case of negative delta_live_tuples */
- tabentry->n_live_tuples = Max(tabentry->n_live_tuples, 0);
- /* Likewise for n_dead_tuples */
- tabentry->n_dead_tuples = Max(tabentry->n_dead_tuples, 0);
-
- /*
- * Add per-table stats to the per-database entry, too.
- */
- dbentry->n_tuples_returned += tabmsg->t_counts.t_tuples_returned;
- dbentry->n_tuples_fetched += tabmsg->t_counts.t_tuples_fetched;
- dbentry->n_tuples_inserted += tabmsg->t_counts.t_tuples_inserted;
- dbentry->n_tuples_updated += tabmsg->t_counts.t_tuples_updated;
- dbentry->n_tuples_deleted += tabmsg->t_counts.t_tuples_deleted;
- dbentry->n_blocks_fetched += tabmsg->t_counts.t_blocks_fetched;
- dbentry->n_blocks_hit += tabmsg->t_counts.t_blocks_hit;
- }
-}
-
-
-/* ----------
- * pgstat_recv_tabpurge() -
- *
- * Arrange for dead table removal.
- * ----------
- */
-static void
-pgstat_recv_tabpurge(PgStat_MsgTabpurge *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
- int i;
-
- dbentry = pgstat_get_db_entry(msg->m_databaseid, false);
-
- /*
- * No need to purge if we don't even know the database.
- */
- if (!dbentry || !dbentry->tables)
- return;
-
- /*
- * Process all table entries in the message.
- */
- for (i = 0; i < msg->m_nentries; i++)
- {
- /* Remove from hashtable if present; we don't care if it's not. */
- (void) hash_search(dbentry->tables,
- (void *) &(msg->m_tableid[i]),
- HASH_REMOVE, NULL);
- }
-}
-
-
-/* ----------
- * pgstat_recv_dropdb() -
- *
- * Arrange for dead database removal
- * ----------
- */
-static void
-pgstat_recv_dropdb(PgStat_MsgDropdb *msg, int len)
-{
- Oid dbid = msg->m_databaseid;
- PgStat_StatDBEntry *dbentry;
-
- /*
- * Lookup the database in the hashtable.
- */
- dbentry = pgstat_get_db_entry(dbid, false);
-
- /*
- * If found, remove it (along with the db statfile).
- */
- if (dbentry)
- {
- char statfile[MAXPGPATH];
-
- get_dbstat_filename(false, false, dbid, statfile, MAXPGPATH);
-
- elog(DEBUG2, "removing stats file \"%s\"", statfile);
- unlink(statfile);
-
- if (dbentry->tables != NULL)
- hash_destroy(dbentry->tables);
- if (dbentry->functions != NULL)
- hash_destroy(dbentry->functions);
-
- if (hash_search(pgStatDBHash,
- (void *) &dbid,
- HASH_REMOVE, NULL) == NULL)
- ereport(ERROR,
- (errmsg("database hash table corrupted during cleanup --- abort")));
- }
-}
-
-
-/* ----------
- * pgstat_recv_resetcounter() -
- *
- * Reset the statistics for the specified database.
- * ----------
- */
-static void
-pgstat_recv_resetcounter(PgStat_MsgResetcounter *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
-
- /*
- * Lookup the database in the hashtable. Nothing to do if not there.
- */
- dbentry = pgstat_get_db_entry(msg->m_databaseid, false);
-
- if (!dbentry)
- return;
-
- /*
- * We simply throw away all the database's table entries by recreating a
- * new hash table for them.
- */
- if (dbentry->tables != NULL)
- hash_destroy(dbentry->tables);
- if (dbentry->functions != NULL)
- hash_destroy(dbentry->functions);
-
- dbentry->tables = NULL;
- dbentry->functions = NULL;
-
- /*
- * Reset database-level stats, too. This creates empty hash tables for
- * tables and functions.
- */
- reset_dbentry_counters(dbentry);
-}
-
-/* ----------
- * pgstat_recv_resetsharedcounter() -
- *
- * Reset some shared statistics of the cluster.
- * ----------
- */
-static void
-pgstat_recv_resetsharedcounter(PgStat_MsgResetsharedcounter *msg, int len)
-{
- if (msg->m_resettarget == RESET_BGWRITER)
- {
- /* Reset the global background writer statistics for the cluster. */
- memset(&globalStats, 0, sizeof(globalStats));
- globalStats.stat_reset_timestamp = GetCurrentTimestamp();
- }
- else if (msg->m_resettarget == RESET_ARCHIVER)
- {
- /* Reset the archiver statistics for the cluster. */
- memset(&archiverStats, 0, sizeof(archiverStats));
- archiverStats.stat_reset_timestamp = GetCurrentTimestamp();
- }
- else if (msg->m_resettarget == RESET_WAL)
- {
- /* Reset the WAL statistics for the cluster. */
- memset(&walStats, 0, sizeof(walStats));
- walStats.stat_reset_timestamp = GetCurrentTimestamp();
- }
-
- /*
- * Presumably the sender of this message validated the target, don't
- * complain here if it's not valid
- */
-}
-
-/* ----------
- * pgstat_recv_resetsinglecounter() -
- *
- * Reset a statistics for a single object
- * ----------
- */
-static void
-pgstat_recv_resetsinglecounter(PgStat_MsgResetsinglecounter *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
-
- dbentry = pgstat_get_db_entry(msg->m_databaseid, false);
-
- if (!dbentry)
- return;
-
- /* Set the reset timestamp for the whole database */
- dbentry->stat_reset_timestamp = GetCurrentTimestamp();
-
- /* Remove object if it exists, ignore it if not */
- if (msg->m_resettype == RESET_TABLE)
- (void) hash_search(dbentry->tables, (void *) &(msg->m_objectid),
- HASH_REMOVE, NULL);
- else if (msg->m_resettype == RESET_FUNCTION)
- (void) hash_search(dbentry->functions, (void *) &(msg->m_objectid),
- HASH_REMOVE, NULL);
-}
-
-/* ----------
- * pgstat_recv_resetslrucounter() -
- *
- * Reset some SLRU statistics of the cluster.
- * ----------
- */
-static void
-pgstat_recv_resetslrucounter(PgStat_MsgResetslrucounter *msg, int len)
-{
- int i;
- TimestampTz ts = GetCurrentTimestamp();
-
- for (i = 0; i < SLRU_NUM_ELEMENTS; i++)
- {
- /* reset entry with the given index, or all entries (index is -1) */
- if ((msg->m_index == -1) || (msg->m_index == i))
- {
- memset(&slruStats[i], 0, sizeof(slruStats[i]));
- slruStats[i].stat_reset_timestamp = ts;
- }
- }
-}
-
-/* ----------
- * pgstat_recv_resetreplslotcounter() -
- *
- * Reset some replication slot statistics of the cluster.
- * ----------
- */
-static void
-pgstat_recv_resetreplslotcounter(PgStat_MsgResetreplslotcounter *msg,
- int len)
-{
- int i;
- int idx = -1;
- TimestampTz ts;
-
- ts = GetCurrentTimestamp();
- if (msg->clearall)
- {
- for (i = 0; i < nReplSlotStats; i++)
- pgstat_reset_replslot(i, ts);
- }
- else
- {
- /* Get the index of replication slot statistics to reset */
- idx = pgstat_replslot_index(msg->m_slotname, false);
-
- /*
- * Nothing to do if the given slot entry is not found. This could
- * happen when the slot with the given name is removed and the
- * corresponding statistics entry is also removed before receiving the
- * reset message.
- */
- if (idx < 0)
- return;
-
- /* Reset the stats for the requested replication slot */
- pgstat_reset_replslot(idx, ts);
- }
-}
-
-
-/* ----------
- * pgstat_recv_autovac() -
- *
- * Process an autovacuum signaling message.
- * ----------
- */
-static void
-pgstat_recv_autovac(PgStat_MsgAutovacStart *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
-
- /*
- * Store the last autovacuum time in the database's hashtable entry.
- */
- dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
-
- dbentry->last_autovac_time = msg->m_start_time;
-}
-
-/* ----------
- * pgstat_recv_vacuum() -
- *
- * Process a VACUUM message.
- * ----------
- */
-static void
-pgstat_recv_vacuum(PgStat_MsgVacuum *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
- PgStat_StatTabEntry *tabentry;
-
- /*
- * Store the data in the table's hashtable entry.
- */
- dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
-
- tabentry = pgstat_get_tab_entry(dbentry, msg->m_tableoid, true);
-
- tabentry->n_live_tuples = msg->m_live_tuples;
- tabentry->n_dead_tuples = msg->m_dead_tuples;
-
- /*
- * It is quite possible that a non-aggressive VACUUM ended up skipping
- * various pages, however, we'll zero the insert counter here regardless.
- * It's currently used only to track when we need to perform an "insert"
- * autovacuum, which are mainly intended to freeze newly inserted tuples.
- * Zeroing this may just mean we'll not try to vacuum the table again
- * until enough tuples have been inserted to trigger another insert
- * autovacuum. An anti-wraparound autovacuum will catch any persistent
- * stragglers.
- */
- tabentry->inserts_since_vacuum = 0;
-
- if (msg->m_autovacuum)
- {
- tabentry->autovac_vacuum_timestamp = msg->m_vacuumtime;
- tabentry->autovac_vacuum_count++;
- }
- else
- {
- tabentry->vacuum_timestamp = msg->m_vacuumtime;
- tabentry->vacuum_count++;
- }
-}
-
-/* ----------
- * pgstat_recv_analyze() -
- *
- * Process an ANALYZE message.
- * ----------
- */
-static void
-pgstat_recv_analyze(PgStat_MsgAnalyze *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
- PgStat_StatTabEntry *tabentry;
-
- /*
- * Store the data in the table's hashtable entry.
- */
- dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
-
- tabentry = pgstat_get_tab_entry(dbentry, msg->m_tableoid, true);
-
- tabentry->n_live_tuples = msg->m_live_tuples;
- tabentry->n_dead_tuples = msg->m_dead_tuples;
-
- /*
- * If commanded, reset changes_since_analyze to zero. This forgets any
- * changes that were committed while the ANALYZE was in progress, but we
- * have no good way to estimate how many of those there were.
- */
- if (msg->m_resetcounter)
- tabentry->changes_since_analyze = 0;
-
- if (msg->m_autovacuum)
- {
- tabentry->autovac_analyze_timestamp = msg->m_analyzetime;
- tabentry->autovac_analyze_count++;
- }
- else
- {
- tabentry->analyze_timestamp = msg->m_analyzetime;
- tabentry->analyze_count++;
- }
-}
-
-
-/* ----------
- * pgstat_recv_archiver() -
- *
- * Process a ARCHIVER message.
- * ----------
- */
-static void
-pgstat_recv_archiver(PgStat_MsgArchiver *msg, int len)
-{
- if (msg->m_failed)
- {
- /* Failed archival attempt */
- ++archiverStats.failed_count;
- memcpy(archiverStats.last_failed_wal, msg->m_xlog,
- sizeof(archiverStats.last_failed_wal));
- archiverStats.last_failed_timestamp = msg->m_timestamp;
- }
- else
- {
- /* Successful archival operation */
- ++archiverStats.archived_count;
- memcpy(archiverStats.last_archived_wal, msg->m_xlog,
- sizeof(archiverStats.last_archived_wal));
- archiverStats.last_archived_timestamp = msg->m_timestamp;
- }
-}
-
-/* ----------
- * pgstat_recv_bgwriter() -
- *
- * Process a BGWRITER message.
- * ----------
- */
-static void
-pgstat_recv_bgwriter(PgStat_MsgBgWriter *msg, int len)
-{
- globalStats.timed_checkpoints += msg->m_timed_checkpoints;
- globalStats.requested_checkpoints += msg->m_requested_checkpoints;
- globalStats.checkpoint_write_time += msg->m_checkpoint_write_time;
- globalStats.checkpoint_sync_time += msg->m_checkpoint_sync_time;
- globalStats.buf_written_checkpoints += msg->m_buf_written_checkpoints;
- globalStats.buf_written_clean += msg->m_buf_written_clean;
- globalStats.maxwritten_clean += msg->m_maxwritten_clean;
- globalStats.buf_written_backend += msg->m_buf_written_backend;
- globalStats.buf_fsync_backend += msg->m_buf_fsync_backend;
- globalStats.buf_alloc += msg->m_buf_alloc;
-}
-
-/* ----------
- * pgstat_recv_wal() -
- *
- * Process a WAL message.
- * ----------
- */
-static void
-pgstat_recv_wal(PgStat_MsgWal *msg, int len)
-{
- walStats.wal_records += msg->m_wal_records;
- walStats.wal_fpi += msg->m_wal_fpi;
- walStats.wal_bytes += msg->m_wal_bytes;
- walStats.wal_buffers_full += msg->m_wal_buffers_full;
- walStats.wal_write += msg->m_wal_write;
- walStats.wal_sync += msg->m_wal_sync;
- walStats.wal_write_time += msg->m_wal_write_time;
- walStats.wal_sync_time += msg->m_wal_sync_time;
-}
-
-/* ----------
- * pgstat_recv_slru() -
- *
- * Process a SLRU message.
- * ----------
- */
-static void
-pgstat_recv_slru(PgStat_MsgSLRU *msg, int len)
-{
- slruStats[msg->m_index].blocks_zeroed += msg->m_blocks_zeroed;
- slruStats[msg->m_index].blocks_hit += msg->m_blocks_hit;
- slruStats[msg->m_index].blocks_read += msg->m_blocks_read;
- slruStats[msg->m_index].blocks_written += msg->m_blocks_written;
- slruStats[msg->m_index].blocks_exists += msg->m_blocks_exists;
- slruStats[msg->m_index].flush += msg->m_flush;
- slruStats[msg->m_index].truncate += msg->m_truncate;
-}
-
-/* ----------
- * pgstat_recv_recoveryconflict() -
- *
- * Process a RECOVERYCONFLICT message.
- * ----------
- */
-static void
-pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
-
- dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
-
- switch (msg->m_reason)
- {
- case PROCSIG_RECOVERY_CONFLICT_DATABASE:
-
- /*
- * Since we drop the information about the database as soon as it
- * replicates, there is no point in counting these conflicts.
- */
- break;
- case PROCSIG_RECOVERY_CONFLICT_TABLESPACE:
- dbentry->n_conflict_tablespace++;
- break;
- case PROCSIG_RECOVERY_CONFLICT_LOCK:
- dbentry->n_conflict_lock++;
- break;
- case PROCSIG_RECOVERY_CONFLICT_SNAPSHOT:
- dbentry->n_conflict_snapshot++;
- break;
- case PROCSIG_RECOVERY_CONFLICT_BUFFERPIN:
- dbentry->n_conflict_bufferpin++;
- break;
- case PROCSIG_RECOVERY_CONFLICT_STARTUP_DEADLOCK:
- dbentry->n_conflict_startup_deadlock++;
- break;
- }
-}
-
-/* ----------
- * pgstat_recv_deadlock() -
- *
- * Process a DEADLOCK message.
- * ----------
- */
-static void
-pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
-
- dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
-
- dbentry->n_deadlocks++;
-}
-
-/* ----------
- * pgstat_recv_checksum_failure() -
- *
- * Process a CHECKSUMFAILURE message.
- * ----------
- */
-static void
-pgstat_recv_checksum_failure(PgStat_MsgChecksumFailure *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
-
- dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
-
- dbentry->n_checksum_failures += msg->m_failurecount;
- dbentry->last_checksum_failure = msg->m_failure_time;
-}
-
-/* ----------
- * pgstat_recv_replslot() -
- *
- * Process a REPLSLOT message.
- * ----------
- */
-static void
-pgstat_recv_replslot(PgStat_MsgReplSlot *msg, int len)
-{
- int idx;
-
- /*
- * Get the index of replication slot statistics. On dropping, we don't
- * create the new statistics.
- */
- idx = pgstat_replslot_index(msg->m_slotname, !msg->m_drop);
-
- /*
- * The slot entry is not found or there is no space to accommodate the new
- * entry. This could happen when the message for the creation of a slot
- * reached before the drop message even though the actual operations
- * happen in reverse order. In such a case, the next update of the
- * statistics for the same slot will create the required entry.
- */
- if (idx < 0)
- return;
-
- /* it must be a valid replication slot index */
- Assert(idx < nReplSlotStats);
-
- if (msg->m_drop)
- {
- /* Remove the replication slot statistics with the given name */
- if (idx < nReplSlotStats - 1)
- memcpy(&replSlotStats[idx], &replSlotStats[nReplSlotStats - 1],
- sizeof(PgStat_ReplSlotStats));
- nReplSlotStats--;
- }
- else
- {
- /* Update the replication slot statistics */
- replSlotStats[idx].spill_txns += msg->m_spill_txns;
- replSlotStats[idx].spill_count += msg->m_spill_count;
- replSlotStats[idx].spill_bytes += msg->m_spill_bytes;
- replSlotStats[idx].stream_txns += msg->m_stream_txns;
- replSlotStats[idx].stream_count += msg->m_stream_count;
- replSlotStats[idx].stream_bytes += msg->m_stream_bytes;
- }
-}
-
-/* ----------
- * pgstat_recv_connstat() -
- *
- * Process connection information.
- * ----------
- */
-static void
-pgstat_recv_connstat(PgStat_MsgConn *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
-
- dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
-
- dbentry->n_sessions += msg->m_count;
- dbentry->total_session_time += msg->m_session_time;
- dbentry->total_active_time += msg->m_active_time;
- dbentry->total_idle_in_xact_time += msg->m_idle_in_xact_time;
- switch (msg->m_disconnect)
- {
- case DISCONNECT_NOT_YET:
- case DISCONNECT_NORMAL:
- /* we don't collect these */
- break;
- case DISCONNECT_CLIENT_EOF:
- dbentry->n_sessions_abandoned++;
- break;
- case DISCONNECT_FATAL:
- dbentry->n_sessions_fatal++;
- break;
- case DISCONNECT_KILLED:
- dbentry->n_sessions_killed++;
- break;
- }
-}
-
-/* ----------
- * pgstat_recv_tempfile() -
- *
- * Process a TEMPFILE message.
- * ----------
- */
-static void
-pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
-
- dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
-
- dbentry->n_temp_bytes += msg->m_filesize;
- dbentry->n_temp_files += 1;
-}
-
-/* ----------
- * pgstat_recv_funcstat() -
- *
- * Count what the backend has done.
- * ----------
- */
-static void
-pgstat_recv_funcstat(PgStat_MsgFuncstat *msg, int len)
-{
- PgStat_FunctionEntry *funcmsg = &(msg->m_entry[0]);
- PgStat_StatDBEntry *dbentry;
- PgStat_StatFuncEntry *funcentry;
- int i;
- bool found;
-
- dbentry = pgstat_get_db_entry(msg->m_databaseid, true);
-
- /*
- * Process all function entries in the message.
- */
- for (i = 0; i < msg->m_nentries; i++, funcmsg++)
- {
- funcentry = (PgStat_StatFuncEntry *) hash_search(dbentry->functions,
- (void *) &(funcmsg->f_id),
- HASH_ENTER, &found);
-
- if (!found)
- {
- /*
- * If it's a new function entry, initialize counters to the values
- * we just got.
- */
- funcentry->f_numcalls = funcmsg->f_numcalls;
- funcentry->f_total_time = funcmsg->f_total_time;
- funcentry->f_self_time = funcmsg->f_self_time;
- }
- else
- {
- /*
- * Otherwise add the values to the existing entry.
- */
- funcentry->f_numcalls += funcmsg->f_numcalls;
- funcentry->f_total_time += funcmsg->f_total_time;
- funcentry->f_self_time += funcmsg->f_self_time;
- }
- }
-}
-
-/* ----------
- * pgstat_recv_funcpurge() -
- *
- * Arrange for dead function removal.
- * ----------
- */
-static void
-pgstat_recv_funcpurge(PgStat_MsgFuncpurge *msg, int len)
-{
- PgStat_StatDBEntry *dbentry;
- int i;
-
- dbentry = pgstat_get_db_entry(msg->m_databaseid, false);
-
- /*
- * No need to purge if we don't even know the database.
- */
- if (!dbentry || !dbentry->functions)
- return;
-
- /*
- * Process all function entries in the message.
- */
- for (i = 0; i < msg->m_nentries; i++)
- {
- /* Remove from hashtable if present; we don't care if it's not. */
- (void) hash_search(dbentry->functions,
- (void *) &(msg->m_functionid[i]),
- HASH_REMOVE, NULL);
- }
-}
-
-/* ----------
- * pgstat_write_statsfile_needed() -
- *
- * Do we need to write out any stats files?
- * ----------
- */
-static bool
-pgstat_write_statsfile_needed(void)
-{
- if (pending_write_requests != NIL)
- return true;
-
- /* Everything was written recently */
- return false;
-}
-
-/* ----------
- * pgstat_db_requested() -
- *
- * Checks whether stats for a particular DB need to be written to a file.
- * ----------
- */
-static bool
-pgstat_db_requested(Oid databaseid)
-{
- /*
- * If any requests are outstanding at all, we should write the stats for
- * shared catalogs (the "database" with OID 0). This ensures that
- * backends will see up-to-date stats for shared catalogs, even though
- * they send inquiry messages mentioning only their own DB.
- */
- if (databaseid == InvalidOid && pending_write_requests != NIL)
- return true;
-
- /* Search to see if there's an open request to write this database. */
- if (list_member_oid(pending_write_requests, databaseid))
- return true;
-
- return false;
+ /* Reset variables */
+ pgStatLocalContext = NULL;
+ localBackendStatusTable = NULL;
+ localNumBackends = 0;
+ }
+
+ /* Invalidate the simple cache keys */
+ cached_dbent_key = stathashkey_zero;
+ cached_tabent_key = stathashkey_zero;
+ cached_funcent_key = stathashkey_zero;
+ cached_archiverstats_is_valid = false;
+ cached_bgwriterstats_is_valid = false;
+ cached_checkpointerstats_is_valid = false;
+ cached_walstats_is_valid = false;
+ cached_slrustats_is_valid = false;
+ n_cached_replslotstats = -1;
}
/*
@@ -7295,60 +6062,6 @@ pgstat_clip_activity(const char *raw_activity)
return activity;
}
-/* ----------
- * pgstat_replslot_index
- *
- * Return the index of entry of a replication slot with the given name, or
- * -1 if the slot is not found.
- *
- * create_it tells whether to create the new slot entry if it is not found.
- * ----------
- */
-static int
-pgstat_replslot_index(const char *name, bool create_it)
-{
- int i;
-
- Assert(nReplSlotStats <= max_replication_slots);
- for (i = 0; i < nReplSlotStats; i++)
- {
- if (strcmp(replSlotStats[i].slotname, name) == 0)
- return i; /* found */
- }
-
- /*
- * The slot is not found. We don't want to register the new statistics if
- * the list is already full or the caller didn't request.
- */
- if (i == max_replication_slots || !create_it)
- return -1;
-
- /* Register new slot */
- memset(&replSlotStats[nReplSlotStats], 0, sizeof(PgStat_ReplSlotStats));
- strlcpy(replSlotStats[nReplSlotStats].slotname, name, NAMEDATALEN);
-
- return nReplSlotStats++;
-}
-
-/* ----------
- * pgstat_reset_replslot
- *
- * Reset the replication slot stats at index 'i'.
- * ----------
- */
-static void
-pgstat_reset_replslot(int i, TimestampTz ts)
-{
- /* reset only counters. Don't clear slot name */
- replSlotStats[i].spill_txns = 0;
- replSlotStats[i].spill_count = 0;
- replSlotStats[i].spill_bytes = 0;
- replSlotStats[i].stream_txns = 0;
- replSlotStats[i].stream_count = 0;
- replSlotStats[i].stream_bytes = 0;
- replSlotStats[i].stat_reset_timestamp = ts;
-}
-
/*
* pgstat_slru_index
*
@@ -7393,7 +6106,7 @@ pgstat_slru_name(int slru_idx)
* Returns pointer to entry with counters for given SLRU (based on the name
* stored in SlruCtl as lwlock tranche name).
*/
-static inline PgStat_MsgSLRU *
+static PgStat_SLRUStats *
slru_entry(int slru_idx)
{
/*
@@ -7404,7 +6117,7 @@ slru_entry(int slru_idx)
Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS));
- return &SLRUStats[slru_idx];
+ return &local_SLRUStats[slru_idx];
}
/*
@@ -7414,41 +6127,48 @@ slru_entry(int slru_idx)
void
pgstat_count_slru_page_zeroed(int slru_idx)
{
- slru_entry(slru_idx)->m_blocks_zeroed += 1;
+ slru_entry(slru_idx)->blocks_zeroed += 1;
+ have_slrustats = true;
}
void
pgstat_count_slru_page_hit(int slru_idx)
{
- slru_entry(slru_idx)->m_blocks_hit += 1;
+ slru_entry(slru_idx)->blocks_hit += 1;
+ have_slrustats = true;
}
void
pgstat_count_slru_page_exists(int slru_idx)
{
- slru_entry(slru_idx)->m_blocks_exists += 1;
+ slru_entry(slru_idx)->blocks_exists += 1;
+ have_slrustats = true;
}
void
pgstat_count_slru_page_read(int slru_idx)
{
- slru_entry(slru_idx)->m_blocks_read += 1;
+ slru_entry(slru_idx)->blocks_read += 1;
+ have_slrustats = true;
}
void
pgstat_count_slru_page_written(int slru_idx)
{
- slru_entry(slru_idx)->m_blocks_written += 1;
+ slru_entry(slru_idx)->blocks_written += 1;
+ have_slrustats = true;
}
void
pgstat_count_slru_flush(int slru_idx)
{
- slru_entry(slru_idx)->m_flush += 1;
+ slru_entry(slru_idx)->flush += 1;
+ have_slrustats = true;
}
void
pgstat_count_slru_truncate(int slru_idx)
{
- slru_entry(slru_idx)->m_truncate += 1;
+ slru_entry(slru_idx)->truncate += 1;
+ have_slrustats = true;
}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index d9f8d82650..0f3b8bbbc7 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -251,7 +251,6 @@ static pid_t StartupPID = 0,
WalReceiverPID = 0,
AutoVacPID = 0,
PgArchPID = 0,
- PgStatPID = 0,
SysLoggerPID = 0;
/* Startup process's status */
@@ -512,7 +511,6 @@ typedef struct
PGPROC *AuxiliaryProcs;
PGPROC *PreparedXactProcs;
PMSignalData *PMSignalState;
- InheritableSocket pgStatSock;
pid_t PostmasterPid;
TimestampTz PgStartTime;
TimestampTz PgReloadTime;
@@ -1331,12 +1329,6 @@ PostmasterMain(int argc, char *argv[])
*/
RemovePgTempFiles();
- /*
- * Initialize stats collection subsystem (this does NOT start the
- * collector process!)
- */
- pgstat_init();
-
/*
* Initialize the autovacuum subsystem (again, no process start yet)
*/
@@ -1786,11 +1778,6 @@ ServerLoop(void)
start_autovac_launcher = false; /* signal processed */
}
- /* If we have lost the stats collector, try to start a new one */
- if (PgStatPID == 0 &&
- (pmState == PM_RUN || pmState == PM_HOT_STANDBY))
- PgStatPID = pgstat_start();
-
/* If we have lost the archiver, try to start a new one. */
if (PgArchPID == 0 && PgArchStartupAllowed())
PgArchPID = StartArchiver();
@@ -2679,8 +2666,6 @@ SIGHUP_handler(SIGNAL_ARGS)
signal_child(PgArchPID, SIGHUP);
if (SysLoggerPID != 0)
signal_child(SysLoggerPID, SIGHUP);
- if (PgStatPID != 0)
- signal_child(PgStatPID, SIGHUP);
/* Reload authentication config files too */
if (!load_hba())
@@ -3009,8 +2994,6 @@ reaper(SIGNAL_ARGS)
AutoVacPID = StartAutoVacLauncher();
if (PgArchStartupAllowed() && PgArchPID == 0)
PgArchPID = StartArchiver();
- if (PgStatPID == 0)
- PgStatPID = pgstat_start();
/* workers may be scheduled to start now */
maybe_start_bgworkers();
@@ -3077,13 +3060,6 @@ reaper(SIGNAL_ARGS)
SignalChildren(SIGUSR2);
pmState = PM_SHUTDOWN_2;
-
- /*
- * We can also shut down the stats collector now; there's
- * nothing left for it to do.
- */
- if (PgStatPID != 0)
- signal_child(PgStatPID, SIGQUIT);
}
else
{
@@ -3154,22 +3130,6 @@ reaper(SIGNAL_ARGS)
continue;
}
- /*
- * Was it the statistics collector? If so, just try to start a new
- * one; no need to force reset of the rest of the system. (If fail,
- * we'll try again in future cycles of the main loop.)
- */
- if (pid == PgStatPID)
- {
- PgStatPID = 0;
- if (!EXIT_STATUS_0(exitstatus))
- LogChildExit(LOG, _("statistics collector process"),
- pid, exitstatus);
- if (pmState == PM_RUN || pmState == PM_HOT_STANDBY)
- PgStatPID = pgstat_start();
- continue;
- }
-
/* Was it the system logger? If so, try to start a new one */
if (pid == SysLoggerPID)
{
@@ -3616,22 +3576,6 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
}
- /*
- * Force a power-cycle of the pgstat process too. (This isn't absolutely
- * necessary, but it seems like a good idea for robustness, and it
- * simplifies the state-machine logic in the case where a shutdown request
- * arrives during crash processing.)
- */
- if (PgStatPID != 0 && take_action)
- {
- ereport(DEBUG2,
- (errmsg_internal("sending %s to process %d",
- "SIGQUIT",
- (int) PgStatPID)));
- signal_child(PgStatPID, SIGQUIT);
- allow_immediate_pgstat_restart();
- }
-
/* We do NOT restart the syslogger */
if (Shutdown != ImmediateShutdown)
@@ -3858,8 +3802,6 @@ PostmasterStateMachine(void)
SignalChildren(SIGQUIT);
if (PgArchPID != 0)
signal_child(PgArchPID, SIGQUIT);
- if (PgStatPID != 0)
- signal_child(PgStatPID, SIGQUIT);
}
}
}
@@ -3883,8 +3825,7 @@ PostmasterStateMachine(void)
{
/*
* PM_WAIT_DEAD_END state ends when the BackendList is entirely empty
- * (ie, no dead_end children remain), and the archiver and stats
- * collector are gone too.
+ * (ie, no dead_end children remain), and the archiveris gone too.
*
* The reason we wait for those two is to protect them against a new
* postmaster starting conflicting subprocesses; this isn't an
@@ -3894,8 +3835,7 @@ PostmasterStateMachine(void)
* normal state transition leading up to PM_WAIT_DEAD_END, or during
* FatalError processing.
*/
- if (dlist_is_empty(&BackendList) &&
- PgArchPID == 0 && PgStatPID == 0)
+ if (dlist_is_empty(&BackendList) && PgArchPID == 0)
{
/* These other guys should be dead already */
Assert(StartupPID == 0);
@@ -4096,8 +4036,6 @@ TerminateChildren(int signal)
signal_child(AutoVacPID, signal);
if (PgArchPID != 0)
signal_child(PgArchPID, signal);
- if (PgStatPID != 0)
- signal_child(PgStatPID, signal);
}
/*
@@ -5043,12 +4981,6 @@ SubPostmasterMain(int argc, char *argv[])
StartBackgroundWorker();
}
- if (strcmp(argv[1], "--forkcol") == 0)
- {
- /* Do not want to attach to shared memory */
-
- PgstatCollectorMain(argc, argv); /* does not return */
- }
if (strcmp(argv[1], "--forklog") == 0)
{
/* Do not want to attach to shared memory */
@@ -5161,12 +5093,6 @@ sigusr1_handler(SIGNAL_ARGS)
if (CheckPostmasterSignal(PMSIGNAL_BEGIN_HOT_STANDBY) &&
pmState == PM_RECOVERY && Shutdown == NoShutdown)
{
- /*
- * Likewise, start other special children as needed.
- */
- Assert(PgStatPID == 0);
- PgStatPID = pgstat_start();
-
ereport(LOG,
(errmsg("database system is ready to accept read only connections")));
@@ -6083,7 +6009,6 @@ extern slock_t *ShmemLock;
extern slock_t *ProcStructLock;
extern PGPROC *AuxiliaryProcs;
extern PMSignalData *PMSignalState;
-extern pgsocket pgStatSock;
extern pg_time_t first_syslogger_file_time;
#ifndef WIN32
@@ -6139,8 +6064,6 @@ save_backend_variables(BackendParameters *param, Port *port,
param->AuxiliaryProcs = AuxiliaryProcs;
param->PreparedXactProcs = PreparedXactProcs;
param->PMSignalState = PMSignalState;
- if (!write_inheritable_socket(¶m->pgStatSock, pgStatSock, childPid))
- return false;
param->PostmasterPid = PostmasterPid;
param->PgStartTime = PgStartTime;
@@ -6373,7 +6296,6 @@ restore_backend_variables(BackendParameters *param, Port *port)
AuxiliaryProcs = param->AuxiliaryProcs;
PreparedXactProcs = param->PreparedXactProcs;
PMSignalState = param->PMSignalState;
- read_inheritable_socket(&pgStatSock, ¶m->pgStatSock);
PostmasterPid = param->PostmasterPid;
PgStartTime = param->PgStartTime;
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 132df29aba..5efd3fb51f 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -254,7 +254,7 @@ WalWriterMain(void)
left_till_hibernate--;
/* Send WAL statistics to the stats collector */
- pgstat_send_wal(false);
+ pgstat_report_wal();
/*
* Sleep until we are signaled or WalWriterDelay has elapsed. If we
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index 240e902d28..fa7bdbcefa 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -1568,8 +1568,8 @@ is_checksummed_file(const char *fullpath, const char *filename)
*
* If 'missing_ok' is true, will not throw an error if the file is not found.
*
- * If dboid is anything other than InvalidOid then any checksum failures detected
- * will get reported to the stats collector.
+ * If dboid is anything other than InvalidOid then any checksum failures
+ * detected will get reported to the activity stats facility.
*
* Returns true if the file was successfully sent, false if 'missing_ok',
* and the file did not exist.
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 75a087c2f9..1f4ba2cafc 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -700,14 +700,10 @@ ReplicationSlotDropPtr(ReplicationSlot *slot)
(errmsg("could not remove directory \"%s\"", tmppath)));
/*
- * Send a message to drop the replication slot to the stats collector.
- * Since there is no guarantee of the order of message transfer on a UDP
- * connection, it's possible that a message for creating a new slot
- * reaches before a message for removing the old slot. We send the drop
- * and create messages while holding ReplicationSlotAllocationLock to
- * reduce that possibility. If the messages reached in reverse, we would
- * lose one statistics update message. But the next update message will
- * create the statistics for the replication slot.
+ * Drop the statistics entry for the replication slot. Do this while
+ * holding ReplicationSlotAllocationLock so that we don't drop a statistics
+ * entry for another slot with the same name just created on another
+ * session.
*/
if (SlotIsLogical(slot))
pgstat_report_replslot_drop(NameStr(slot->data.name));
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092..517354bed2 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -2059,7 +2059,7 @@ BufferSync(int flags)
if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
{
TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- BgWriterStats.m_buf_written_checkpoints++;
+ CheckPointerStats.buf_written_checkpoints++;
num_written++;
}
}
@@ -2169,7 +2169,7 @@ BgBufferSync(WritebackContext *wb_context)
strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
/* Report buffer alloc counts to pgstat */
- BgWriterStats.m_buf_alloc += recent_alloc;
+ BgWriterStats.buf_alloc += recent_alloc;
/*
* If we're not running the LRU scan, just stop after doing the stats
@@ -2359,7 +2359,7 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
- BgWriterStats.m_maxwritten_clean++;
+ BgWriterStats.maxwritten_clean++;
break;
}
}
@@ -2367,7 +2367,7 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
- BgWriterStats.m_buf_written_clean += num_written;
+ BgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index f9bbe97b50..78cfe91eab 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -149,6 +149,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, BTreeShmemSize());
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
+ size = add_size(size, StatsShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -267,6 +268,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ StatsShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 8cb6a6f042..bb202dbc2b 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -177,7 +177,9 @@ static const char *const BuiltinTrancheNames[] = {
/* LWTRANCHE_PARALLEL_APPEND: */
"ParallelAppend",
/* LWTRANCHE_PER_XACT_PREDICATE_LIST: */
- "PerXactPredicateList"
+ "PerXactPredicateList",
+ /* LWTRANCHE_STATS: */
+ "ActivityStatistics"
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 6c7cf6c295..8edb41c1cf 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+StatsLock 48
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 4dc24649df..75d1695576 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -414,8 +414,8 @@ smgrdounlinkall(SMgrRelation *rels, int nrels, bool isRedo)
}
/*
- * It'd be nice to tell the stats collector to forget them immediately,
- * too. But we can't because we don't know the OIDs.
+ * It'd be nice to tell the activity stats facility to forget them
+ * immediately, too. But we can't because we don't know the OIDs.
*/
/*
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 8a0332dde9..3e3a48111c 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3207,6 +3207,12 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (IdleStatsUpdateTimeoutPending)
+ {
+ IdleStatsUpdateTimeoutPending = false;
+ pgstat_report_stat(true);
+ }
}
@@ -3778,6 +3784,7 @@ PostgresMain(int argc, char *argv[],
volatile bool send_ready_for_query = true;
bool idle_in_transaction_timeout_enabled = false;
bool idle_session_timeout_enabled = false;
+ bool idle_stats_update_timeout_enabled = false;
/* Initialize startup process environment if necessary. */
if (!IsUnderPostmaster)
@@ -4174,11 +4181,12 @@ PostgresMain(int argc, char *argv[],
* Note: this includes fflush()'ing the last of the prior output.
*
* This is also a good time to send collected statistics to the
- * collector, and to update the PS stats display. We avoid doing
- * those every time through the message loop because it'd slow down
- * processing of batched messages, and because we don't want to report
- * uncommitted updates (that confuses autovacuum). The notification
- * processor wants a call too, if we are not in a transaction block.
+ * activity statistics, and to update the PS stats display. We avoid
+ * doing those every time through the message loop because it'd slow
+ * down processing of batched messages, and because we don't want to
+ * report uncommitted updates (that confuses autovacuum). The
+ * notification processor wants a call too, if we are not in a
+ * transaction block.
*
* Also, if an idle timeout is enabled, start the timer for that.
*/
@@ -4212,6 +4220,8 @@ PostgresMain(int argc, char *argv[],
}
else
{
+ long stats_timeout;
+
/* Send out notify signals and transmit self-notifies */
ProcessCompletedNotifies();
@@ -4224,8 +4234,14 @@ PostgresMain(int argc, char *argv[],
if (notifyInterruptPending)
ProcessNotifyInterrupt();
- pgstat_report_stat(false);
-
+ /* Start the idle-stats-update timer */
+ stats_timeout = pgstat_report_stat(false);
+ if (stats_timeout > 0)
+ {
+ idle_stats_update_timeout_enabled = true;
+ enable_timeout_after(IDLE_STATS_UPDATE_TIMEOUT,
+ stats_timeout);
+ }
set_ps_display("idle");
pgstat_report_activity(STATE_IDLE, NULL);
@@ -4259,9 +4275,9 @@ PostgresMain(int argc, char *argv[],
firstchar = ReadCommand(&input_message);
/*
- * (4) turn off the idle-in-transaction and idle-session timeouts, if
- * active. We do this before step (5) so that any last-moment timeout
- * is certain to be detected in step (5).
+ * (4) turn off the idle-in-transaction, idle-session and
+ * idle-state-update timeouts if active. We do this before step (5) so
+ * that any last-moment timeout is certain to be detected in step (5).
*
* At most one of these timeouts will be active, so there's no need to
* worry about combining the timeout.c calls into one.
@@ -4276,6 +4292,11 @@ PostgresMain(int argc, char *argv[],
disable_timeout(IDLE_SESSION_TIMEOUT, false);
idle_session_timeout_enabled = false;
}
+ if (idle_stats_update_timeout_enabled)
+ {
+ disable_timeout(IDLE_STATS_UPDATE_TIMEOUT, false);
+ idle_stats_update_timeout_enabled = false;
+ }
/*
* (5) disable async signal conditions again.
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 5102227a60..daa062b603 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1,7 +1,7 @@
/*-------------------------------------------------------------------------
*
* pgstatfuncs.c
- * Functions for accessing the statistics collector data
+ * Functions for accessing the activity statistics data
*
* Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -35,9 +35,6 @@
#define HAS_PGSTAT_PERMISSIONS(role) (is_member_of_role(GetUserId(), DEFAULT_ROLE_READ_ALL_STATS) || has_privs_of_role(GetUserId(), role))
-/* Global bgwriter statistics, from bgwriter.c */
-extern PgStat_MsgBgWriter bgwriterStats;
-
Datum
pg_stat_get_numscans(PG_FUNCTION_ARGS)
{
@@ -1267,7 +1264,7 @@ pg_stat_get_db_xact_commit(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_xact_commit);
+ result = (int64) (dbentry->counts.n_xact_commit);
PG_RETURN_INT64(result);
}
@@ -1283,7 +1280,7 @@ pg_stat_get_db_xact_rollback(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_xact_rollback);
+ result = (int64) (dbentry->counts.n_xact_rollback);
PG_RETURN_INT64(result);
}
@@ -1299,7 +1296,7 @@ pg_stat_get_db_blocks_fetched(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_blocks_fetched);
+ result = (int64) (dbentry->counts.n_blocks_fetched);
PG_RETURN_INT64(result);
}
@@ -1315,7 +1312,7 @@ pg_stat_get_db_blocks_hit(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_blocks_hit);
+ result = (int64) (dbentry->counts.n_blocks_hit);
PG_RETURN_INT64(result);
}
@@ -1331,7 +1328,7 @@ pg_stat_get_db_tuples_returned(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_tuples_returned);
+ result = (int64) (dbentry->counts.n_tuples_returned);
PG_RETURN_INT64(result);
}
@@ -1347,7 +1344,7 @@ pg_stat_get_db_tuples_fetched(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_tuples_fetched);
+ result = (int64) (dbentry->counts.n_tuples_fetched);
PG_RETURN_INT64(result);
}
@@ -1363,7 +1360,7 @@ pg_stat_get_db_tuples_inserted(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_tuples_inserted);
+ result = (int64) (dbentry->counts.n_tuples_inserted);
PG_RETURN_INT64(result);
}
@@ -1379,7 +1376,7 @@ pg_stat_get_db_tuples_updated(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_tuples_updated);
+ result = (int64) (dbentry->counts.n_tuples_updated);
PG_RETURN_INT64(result);
}
@@ -1395,7 +1392,7 @@ pg_stat_get_db_tuples_deleted(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_tuples_deleted);
+ result = (int64) (dbentry->counts.n_tuples_deleted);
PG_RETURN_INT64(result);
}
@@ -1428,7 +1425,7 @@ pg_stat_get_db_temp_files(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = dbentry->n_temp_files;
+ result = dbentry->counts.n_temp_files;
PG_RETURN_INT64(result);
}
@@ -1444,7 +1441,7 @@ pg_stat_get_db_temp_bytes(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = dbentry->n_temp_bytes;
+ result = dbentry->counts.n_temp_bytes;
PG_RETURN_INT64(result);
}
@@ -1459,7 +1456,7 @@ pg_stat_get_db_conflict_tablespace(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_conflict_tablespace);
+ result = (int64) (dbentry->counts.n_conflict_tablespace);
PG_RETURN_INT64(result);
}
@@ -1474,7 +1471,7 @@ pg_stat_get_db_conflict_lock(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_conflict_lock);
+ result = (int64) (dbentry->counts.n_conflict_lock);
PG_RETURN_INT64(result);
}
@@ -1489,7 +1486,7 @@ pg_stat_get_db_conflict_snapshot(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_conflict_snapshot);
+ result = (int64) (dbentry->counts.n_conflict_snapshot);
PG_RETURN_INT64(result);
}
@@ -1504,7 +1501,7 @@ pg_stat_get_db_conflict_bufferpin(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_conflict_bufferpin);
+ result = (int64) (dbentry->counts.n_conflict_bufferpin);
PG_RETURN_INT64(result);
}
@@ -1519,7 +1516,7 @@ pg_stat_get_db_conflict_startup_deadlock(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_conflict_startup_deadlock);
+ result = (int64) (dbentry->counts.n_conflict_startup_deadlock);
PG_RETURN_INT64(result);
}
@@ -1534,11 +1531,11 @@ pg_stat_get_db_conflict_all(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_conflict_tablespace +
- dbentry->n_conflict_lock +
- dbentry->n_conflict_snapshot +
- dbentry->n_conflict_bufferpin +
- dbentry->n_conflict_startup_deadlock);
+ result = (int64) (dbentry->counts.n_conflict_tablespace +
+ dbentry->counts.n_conflict_lock +
+ dbentry->counts.n_conflict_snapshot +
+ dbentry->counts.n_conflict_bufferpin +
+ dbentry->counts.n_conflict_startup_deadlock);
PG_RETURN_INT64(result);
}
@@ -1553,7 +1550,7 @@ pg_stat_get_db_deadlocks(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_deadlocks);
+ result = (int64) (dbentry->counts.n_deadlocks);
PG_RETURN_INT64(result);
}
@@ -1571,7 +1568,7 @@ pg_stat_get_db_checksum_failures(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = (int64) (dbentry->n_checksum_failures);
+ result = (int64) (dbentry->counts.n_checksum_failures);
PG_RETURN_INT64(result);
}
@@ -1608,7 +1605,7 @@ pg_stat_get_db_blk_read_time(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = ((double) dbentry->n_block_read_time) / 1000.0;
+ result = ((double) dbentry->counts.n_block_read_time) / 1000.0;
PG_RETURN_FLOAT8(result);
}
@@ -1624,7 +1621,7 @@ pg_stat_get_db_blk_write_time(PG_FUNCTION_ARGS)
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL)
result = 0;
else
- result = ((double) dbentry->n_block_write_time) / 1000.0;
+ result = ((double) dbentry->counts.n_block_write_time) / 1000.0;
PG_RETURN_FLOAT8(result);
}
@@ -1638,7 +1635,7 @@ pg_stat_get_db_session_time(PG_FUNCTION_ARGS)
/* convert counter from microsec to millisec for display */
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) != NULL)
- result = ((double) dbentry->total_session_time) / 1000.0;
+ result = ((double) dbentry->counts.total_session_time) / 1000.0;
PG_RETURN_FLOAT8(result);
}
@@ -1652,7 +1649,7 @@ pg_stat_get_db_active_time(PG_FUNCTION_ARGS)
/* convert counter from microsec to millisec for display */
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) != NULL)
- result = ((double) dbentry->total_active_time) / 1000.0;
+ result = ((double) dbentry->counts.total_active_time) / 1000.0;
PG_RETURN_FLOAT8(result);
}
@@ -1666,7 +1663,7 @@ pg_stat_get_db_idle_in_transaction_time(PG_FUNCTION_ARGS)
/* convert counter from microsec to millisec for display */
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) != NULL)
- result = ((double) dbentry->total_idle_in_xact_time) / 1000.0;
+ result = ((double) dbentry->counts.total_idle_in_xact_time) / 1000.0;
PG_RETURN_FLOAT8(result);
}
@@ -1679,7 +1676,7 @@ pg_stat_get_db_sessions(PG_FUNCTION_ARGS)
PgStat_StatDBEntry *dbentry;
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) != NULL)
- result = (int64) (dbentry->n_sessions);
+ result = (int64) (dbentry->counts.n_sessions);
PG_RETURN_INT64(result);
}
@@ -1692,7 +1689,7 @@ pg_stat_get_db_sessions_abandoned(PG_FUNCTION_ARGS)
PgStat_StatDBEntry *dbentry;
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) != NULL)
- result = (int64) (dbentry->n_sessions_abandoned);
+ result = (int64) (dbentry->counts.n_sessions_abandoned);
PG_RETURN_INT64(result);
}
@@ -1705,7 +1702,7 @@ pg_stat_get_db_sessions_fatal(PG_FUNCTION_ARGS)
PgStat_StatDBEntry *dbentry;
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) != NULL)
- result = (int64) (dbentry->n_sessions_fatal);
+ result = (int64) (dbentry->counts.n_sessions_fatal);
PG_RETURN_INT64(result);
}
@@ -1718,7 +1715,7 @@ pg_stat_get_db_sessions_killed(PG_FUNCTION_ARGS)
PgStat_StatDBEntry *dbentry;
if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) != NULL)
- result = (int64) (dbentry->n_sessions_killed);
+ result = (int64) (dbentry->counts.n_sessions_killed);
PG_RETURN_INT64(result);
}
@@ -1726,69 +1723,71 @@ pg_stat_get_db_sessions_killed(PG_FUNCTION_ARGS)
Datum
pg_stat_get_bgwriter_timed_checkpoints(PG_FUNCTION_ARGS)
{
- PG_RETURN_INT64(pgstat_fetch_global()->timed_checkpoints);
+ PG_RETURN_INT64(pgstat_fetch_stat_checkpointer()->timed_checkpoints);
}
Datum
pg_stat_get_bgwriter_requested_checkpoints(PG_FUNCTION_ARGS)
{
- PG_RETURN_INT64(pgstat_fetch_global()->requested_checkpoints);
+ PG_RETURN_INT64(pgstat_fetch_stat_checkpointer()->requested_checkpoints);
}
Datum
pg_stat_get_bgwriter_buf_written_checkpoints(PG_FUNCTION_ARGS)
{
- PG_RETURN_INT64(pgstat_fetch_global()->buf_written_checkpoints);
+ PG_RETURN_INT64(pgstat_fetch_stat_checkpointer()->buf_written_checkpoints);
}
Datum
pg_stat_get_bgwriter_buf_written_clean(PG_FUNCTION_ARGS)
{
- PG_RETURN_INT64(pgstat_fetch_global()->buf_written_clean);
+ PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_written_clean);
}
Datum
pg_stat_get_bgwriter_maxwritten_clean(PG_FUNCTION_ARGS)
{
- PG_RETURN_INT64(pgstat_fetch_global()->maxwritten_clean);
+ PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->maxwritten_clean);
}
Datum
pg_stat_get_checkpoint_write_time(PG_FUNCTION_ARGS)
{
/* time is already in msec, just convert to double for presentation */
- PG_RETURN_FLOAT8((double) pgstat_fetch_global()->checkpoint_write_time);
+ PG_RETURN_FLOAT8((double)
+ pgstat_fetch_stat_checkpointer()->checkpoint_write_time);
}
Datum
pg_stat_get_checkpoint_sync_time(PG_FUNCTION_ARGS)
{
/* time is already in msec, just convert to double for presentation */
- PG_RETURN_FLOAT8((double) pgstat_fetch_global()->checkpoint_sync_time);
+ PG_RETURN_FLOAT8((double)
+ pgstat_fetch_stat_checkpointer()->checkpoint_sync_time);
}
Datum
pg_stat_get_bgwriter_stat_reset_time(PG_FUNCTION_ARGS)
{
- PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->stat_reset_timestamp);
+ PG_RETURN_TIMESTAMPTZ(pgstat_fetch_stat_bgwriter()->stat_reset_timestamp);
}
Datum
pg_stat_get_buf_written_backend(PG_FUNCTION_ARGS)
{
- PG_RETURN_INT64(pgstat_fetch_global()->buf_written_backend);
+ PG_RETURN_INT64(pgstat_fetch_stat_checkpointer()->buf_written_backend);
}
Datum
pg_stat_get_buf_fsync_backend(PG_FUNCTION_ARGS)
{
- PG_RETURN_INT64(pgstat_fetch_global()->buf_fsync_backend);
+ PG_RETURN_INT64(pgstat_fetch_stat_checkpointer()->buf_fsync_backend);
}
Datum
pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
{
- PG_RETURN_INT64(pgstat_fetch_global()->buf_alloc);
+ PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
/*
@@ -1802,7 +1801,7 @@ pg_stat_get_wal(PG_FUNCTION_ARGS)
Datum values[PG_STAT_GET_WAL_COLS];
bool nulls[PG_STAT_GET_WAL_COLS];
char buf[256];
- PgStat_WalStats *wal_stats;
+ PgStat_Wal *wal_stats;
/* Initialise values and NULL flags arrays */
MemSet(values, 0, sizeof(values));
@@ -1835,11 +1834,11 @@ pg_stat_get_wal(PG_FUNCTION_ARGS)
wal_stats = pgstat_fetch_stat_wal();
/* Fill values and NULLs */
- values[0] = Int64GetDatum(wal_stats->wal_records);
- values[1] = Int64GetDatum(wal_stats->wal_fpi);
+ values[0] = Int64GetDatum(wal_stats->wal_usage.wal_records);
+ values[1] = Int64GetDatum(wal_stats->wal_usage.wal_fpi);
/* Convert to numeric. */
- snprintf(buf, sizeof buf, UINT64_FORMAT, wal_stats->wal_bytes);
+ snprintf(buf, sizeof buf, UINT64_FORMAT, wal_stats->wal_usage.wal_bytes);
values[2] = DirectFunctionCall3(numeric_in,
CStringGetDatum(buf),
ObjectIdGetDatum(0),
@@ -2127,7 +2126,7 @@ pg_stat_get_xact_function_self_time(PG_FUNCTION_ARGS)
Datum
pg_stat_get_snapshot_timestamp(PG_FUNCTION_ARGS)
{
- PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->stats_timestamp);
+ PG_RETURN_TIMESTAMPTZ(pgstat_get_stat_timestamp());
}
/* Discard the active statistics snapshot */
@@ -2215,7 +2214,7 @@ pg_stat_get_archiver(PG_FUNCTION_ARGS)
TupleDesc tupdesc;
Datum values[7];
bool nulls[7];
- PgStat_ArchiverStats *archiver_stats;
+ PgStat_Archiver *archiver_stats;
/* Initialise values and NULL flags arrays */
MemSet(values, 0, sizeof(values));
@@ -2285,7 +2284,7 @@ pg_stat_get_replication_slots(PG_FUNCTION_ARGS)
Tuplestorestate *tupstore;
MemoryContext per_query_ctx;
MemoryContext oldcontext;
- PgStat_ReplSlotStats *slotstats;
+ PgStat_ReplSlot *slotstats;
int nstats;
int i;
@@ -2318,7 +2317,7 @@ pg_stat_get_replication_slots(PG_FUNCTION_ARGS)
{
Datum values[PG_STAT_GET_REPLICATION_SLOT_COLS];
bool nulls[PG_STAT_GET_REPLICATION_SLOT_COLS];
- PgStat_ReplSlotStats *s = &(slotstats[i]);
+ PgStat_ReplSlot *s = &(slotstats[i]);
MemSet(values, 0, sizeof(values));
MemSet(nulls, 0, sizeof(nulls));
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..0762c2034c 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -72,6 +72,7 @@
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
#include "optimizer/optimizer.h"
+#include "pgstat.h"
#include "rewrite/rewriteDefine.h"
#include "rewrite/rowsecurity.h"
#include "storage/lmgr.h"
@@ -2366,6 +2367,10 @@ RelationDestroyRelation(Relation relation, bool remember_tupdesc)
*/
RelationCloseSmgr(relation);
+ /* break mutual link with stats entry */
+ pgstat_delinkstats(relation);
+
+ if (relation->rd_rel)
/*
* Free all the subsidiary data structures of the relcache entry, then the
* entry itself.
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index a5976ad5b1..a207ea2d59 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -34,6 +34,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 8b73850d0d..5c32d747cf 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -271,9 +271,6 @@ GetBackendTypeDesc(BackendType backendType)
case B_ARCHIVER:
backendDesc = "archiver";
break;
- case B_STATS_COLLECTOR:
- backendDesc = "stats collector";
- break;
case B_LOGGER:
backendDesc = "logger";
break;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 7abeccb536..32a22aaa24 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -73,6 +73,7 @@ static void StatementTimeoutHandler(void);
static void LockTimeoutHandler(void);
static void IdleInTransactionSessionTimeoutHandler(void);
static void IdleSessionTimeoutHandler(void);
+static void IdleStatsUpdateTimeoutHandler(void);
static bool ThereIsAtLeastOneRole(void);
static void process_startup_options(Port *port, bool am_superuser);
static void process_settings(Oid databaseid, Oid roleid);
@@ -620,6 +621,8 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
RegisterTimeout(IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
IdleInTransactionSessionTimeoutHandler);
RegisterTimeout(IDLE_SESSION_TIMEOUT, IdleSessionTimeoutHandler);
+ RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT,
+ IdleStatsUpdateTimeoutHandler);
}
/*
@@ -1242,6 +1245,14 @@ IdleSessionTimeoutHandler(void)
SetLatch(MyLatch);
}
+static void
+IdleStatsUpdateTimeoutHandler(void)
+{
+ IdleStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
+
/*
* Returns true if at least one role is defined in this database cluster.
*/
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index e337df42cb..d42aea7324 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -746,8 +746,8 @@ const char *const config_group_names[] =
gettext_noop("Statistics"),
/* STATS_MONITORING */
gettext_noop("Statistics / Monitoring"),
- /* STATS_COLLECTOR */
- gettext_noop("Statistics / Query and Index Statistics Collector"),
+ /* STATS_ACTIVITY */
+ gettext_noop("Statistics / Query and Index Activity Statistics"),
/* AUTOVACUUM */
gettext_noop("Autovacuum"),
/* CLIENT_CONN */
@@ -1457,7 +1457,7 @@ static struct config_bool ConfigureNamesBool[] =
#endif
{
- {"track_activities", PGC_SUSET, STATS_COLLECTOR,
+ {"track_activities", PGC_SUSET, STATS_ACTIVITY,
gettext_noop("Collects information about executing commands."),
gettext_noop("Enables the collection of information on the currently "
"executing command of each session, along with "
@@ -1468,7 +1468,7 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
{
- {"track_counts", PGC_SUSET, STATS_COLLECTOR,
+ {"track_counts", PGC_SUSET, STATS_ACTIVITY,
gettext_noop("Collects statistics on database activity."),
NULL
},
@@ -1477,7 +1477,7 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
{
- {"track_io_timing", PGC_SUSET, STATS_COLLECTOR,
+ {"track_io_timing", PGC_SUSET, STATS_ACTIVITY,
gettext_noop("Collects timing statistics for database I/O activity."),
NULL
},
@@ -1486,7 +1486,7 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
{
- {"track_wal_io_timing", PGC_SUSET, STATS_COLLECTOR,
+ {"track_wal_io_timing", PGC_SUSET, STATS_ACTIVITY,
gettext_noop("Collects timing statistics for WAL I/O activity."),
NULL
},
@@ -4376,7 +4376,7 @@ static struct config_string ConfigureNamesString[] =
},
{
- {"stats_temp_directory", PGC_SIGHUP, STATS_COLLECTOR,
+ {"stats_temp_directory", PGC_SIGHUP, STATS_ACTIVITY,
gettext_noop("Writes temporary statistics files to the specified directory."),
NULL,
GUC_SUPERUSER_ONLY
@@ -4712,7 +4712,7 @@ static struct config_enum ConfigureNamesEnum[] =
},
{
- {"track_functions", PGC_SUSET, STATS_COLLECTOR,
+ {"track_functions", PGC_SUSET, STATS_ACTIVITY,
gettext_noop("Collects function-level statistics on database activity."),
NULL
},
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c6483fa1ff..e543f5560f 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -581,7 +581,7 @@
# STATISTICS
#------------------------------------------------------------------------------
-# - Query and Index Statistics Collector -
+# - Query and Index Activity Statistics -
#track_activities = on
#track_counts = on
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index 9eba7d8d7d..fd12dc4c13 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -6,7 +6,7 @@ use File::Basename qw(basename dirname);
use File::Path qw(rmtree);
use PostgresNode;
use TestLib;
-use Test::More tests => 109;
+use Test::More tests => 108;
program_help_ok('pg_basebackup');
program_version_ok('pg_basebackup');
@@ -124,7 +124,7 @@ is_deeply(
# Contents of these directories should not be copied.
foreach my $dirname (
- qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_stat_tmp pg_subtrans)
+ qw(pg_dynshmem pg_notify pg_replslot pg_serial pg_snapshots pg_subtrans)
)
{
is_deeply(
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index adb9f819bb..12708b9470 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -84,6 +84,8 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t ConfigReloadPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
@@ -322,7 +324,6 @@ typedef enum BackendType
B_WAL_SENDER,
B_WAL_WRITER,
B_ARCHIVER,
- B_STATS_COLLECTOR,
B_LOGGER,
} BackendType;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 2ccd60f38c..44ef3fdc35 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -1,7 +1,7 @@
/* ----------
* pgstat.h
*
- * Definitions for the PostgreSQL statistics collector daemon.
+ * Definitions for the PostgreSQL activity statistics facility.
*
* Copyright (c) 2001-2021, PostgreSQL Global Development Group
*
@@ -12,12 +12,15 @@
#define PGSTAT_H
#include "datatype/timestamp.h"
+#include "executor/instrument.h"
#include "libpq/pqcomm.h"
#include "miscadmin.h"
#include "port/atomics.h"
+#include "lib/dshash.h"
#include "portability/instr_time.h"
#include "postmaster/pgarch.h"
#include "storage/proc.h"
+#include "storage/lwlock.h"
#include "utils/hsearch.h"
#include "utils/relcache.h"
@@ -27,11 +30,11 @@
* ----------
*/
#define PGSTAT_STAT_PERMANENT_DIRECTORY "pg_stat"
-#define PGSTAT_STAT_PERMANENT_FILENAME "pg_stat/global.stat"
-#define PGSTAT_STAT_PERMANENT_TMPFILE "pg_stat/global.tmp"
+#define PGSTAT_STAT_PERMANENT_FILENAME "pg_stat/saved_stats"
+#define PGSTAT_STAT_PERMANENT_TMPFILE "pg_stat/saved_stats.tmp"
/* Default directory to store temporary statistics data in */
-#define PG_STAT_TMP_DIR "pg_stat_tmp"
+#define PG_STAT_TMP_DIR "pg_stat_tmp"
/* Values for track_functions GUC variable --- order is significant! */
typedef enum TrackFunctionsLevel
@@ -51,39 +54,6 @@ typedef enum SessionEndType
DISCONNECT_KILLED
} SessionEndType;
-/* ----------
- * The types of backend -> collector messages
- * ----------
- */
-typedef enum StatMsgType
-{
- PGSTAT_MTYPE_DUMMY,
- PGSTAT_MTYPE_INQUIRY,
- PGSTAT_MTYPE_TABSTAT,
- PGSTAT_MTYPE_TABPURGE,
- PGSTAT_MTYPE_DROPDB,
- PGSTAT_MTYPE_RESETCOUNTER,
- PGSTAT_MTYPE_RESETSHAREDCOUNTER,
- PGSTAT_MTYPE_RESETSINGLECOUNTER,
- PGSTAT_MTYPE_RESETSLRUCOUNTER,
- PGSTAT_MTYPE_RESETREPLSLOTCOUNTER,
- PGSTAT_MTYPE_AUTOVAC_START,
- PGSTAT_MTYPE_VACUUM,
- PGSTAT_MTYPE_ANALYZE,
- PGSTAT_MTYPE_ARCHIVER,
- PGSTAT_MTYPE_BGWRITER,
- PGSTAT_MTYPE_WAL,
- PGSTAT_MTYPE_SLRU,
- PGSTAT_MTYPE_FUNCSTAT,
- PGSTAT_MTYPE_FUNCPURGE,
- PGSTAT_MTYPE_RECOVERYCONFLICT,
- PGSTAT_MTYPE_TEMPFILE,
- PGSTAT_MTYPE_DEADLOCK,
- PGSTAT_MTYPE_CHECKSUMFAILURE,
- PGSTAT_MTYPE_REPLSLOT,
- PGSTAT_MTYPE_CONNECTION,
-} StatMsgType;
-
/* ----------
* The data type used for counters.
* ----------
@@ -94,9 +64,8 @@ typedef int64 PgStat_Counter;
* PgStat_TableCounts The actual per-table counts kept by a backend
*
* This struct should contain only actual event counters, because we memcmp
- * it against zeroes to detect whether there are any counts to transmit.
- * It is a component of PgStat_TableStatus (within-backend state) and
- * PgStat_TableEntry (the transmitted message format).
+ * it against zeroes to detect whether there are any counts to write.
+ * It is a component of PgStat_TableStatus (within-backend state).
*
* Note: for a table, tuples_returned is the number of tuples successfully
* fetched by heap_getnext, while tuples_fetched is the number of tuples
@@ -170,10 +139,13 @@ typedef enum PgStat_Single_Reset_Type
*/
typedef struct PgStat_TableStatus
{
- Oid t_id; /* table's OID */
- bool t_shared; /* is it a shared catalog? */
struct PgStat_TableXactStatus *trans; /* lowest subxact's counts */
+ TimestampTz vacuum_timestamp; /* user initiated vacuum */
+ TimestampTz autovac_vacuum_timestamp; /* autovacuum initiated */
+ TimestampTz analyze_timestamp; /* user initiated */
+ TimestampTz autovac_analyze_timestamp; /* autovacuum initiated */
PgStat_TableCounts t_counts; /* event counts to be sent */
+ Relation relation; /* rel that is using this entry */
} PgStat_TableStatus;
/* ----------
@@ -197,359 +169,57 @@ typedef struct PgStat_TableXactStatus
struct PgStat_TableXactStatus *next; /* next of same subxact */
} PgStat_TableXactStatus;
-
-/* ------------------------------------------------------------
- * Message formats follow
- * ------------------------------------------------------------
- */
-
-
-/* ----------
- * PgStat_MsgHdr The common message header
- * ----------
+/*
+ * Archiver statistics kept in the shared stats
*/
-typedef struct PgStat_MsgHdr
+typedef struct PgStat_Archiver
{
- StatMsgType m_type;
- int m_size;
-} PgStat_MsgHdr;
-
-/* ----------
- * Space available in a message. This will keep the UDP packets below 1K,
- * which should fit unfragmented into the MTU of the loopback interface.
- * (Larger values of PGSTAT_MAX_MSG_SIZE would work for that on most
- * platforms, but we're being conservative here.)
- * ----------
- */
-#define PGSTAT_MAX_MSG_SIZE 1000
-#define PGSTAT_MSG_PAYLOAD (PGSTAT_MAX_MSG_SIZE - sizeof(PgStat_MsgHdr))
-
+ PgStat_Counter archived_count; /* archival successes */
+ char last_archived_wal[MAX_XFN_CHARS + 1]; /* last WAL file
+ * archived */
+ TimestampTz last_archived_timestamp; /* last archival success time */
+ PgStat_Counter failed_count; /* failed archival attempts */
+ char last_failed_wal[MAX_XFN_CHARS + 1]; /* WAL file involved in
+ * last failure */
+ TimestampTz last_failed_timestamp; /* last archival failure time */
+ TimestampTz stat_reset_timestamp;
+} PgStat_Archiver;
/* ----------
- * PgStat_MsgDummy A dummy message, ignored by the collector
+ * PgStat_BgWriter bgwriter statistics
* ----------
*/
-typedef struct PgStat_MsgDummy
+typedef struct PgStat_BgWriter
{
- PgStat_MsgHdr m_hdr;
-} PgStat_MsgDummy;
-
+ PgStat_Counter buf_written_clean;
+ PgStat_Counter maxwritten_clean;
+ PgStat_Counter buf_alloc;
+ TimestampTz stat_reset_timestamp;
+} PgStat_BgWriter;
/* ----------
- * PgStat_MsgInquiry Sent by a backend to ask the collector
- * to write the stats file(s).
- *
- * Ordinarily, an inquiry message prompts writing of the global stats file,
- * the stats file for shared catalogs, and the stats file for the specified
- * database. If databaseid is InvalidOid, only the first two are written.
- *
- * New file(s) will be written only if the existing file has a timestamp
- * older than the specified cutoff_time; this prevents duplicated effort
- * when multiple requests arrive at nearly the same time, assuming that
- * backends send requests with cutoff_times a little bit in the past.
- *
- * clock_time should be the requestor's current local time; the collector
- * uses this to check for the system clock going backward, but it has no
- * effect unless that occurs. We assume clock_time >= cutoff_time, though.
+ * PgStat_CheckPointer checkpointer statistics
* ----------
*/
-
-typedef struct PgStat_MsgInquiry
+typedef struct PgStat_CheckPointer
{
- PgStat_MsgHdr m_hdr;
- TimestampTz clock_time; /* observed local clock time */
- TimestampTz cutoff_time; /* minimum acceptable file timestamp */
- Oid databaseid; /* requested DB (InvalidOid => shared only) */
-} PgStat_MsgInquiry;
-
-
-/* ----------
- * PgStat_TableEntry Per-table info in a MsgTabstat
- * ----------
- */
-typedef struct PgStat_TableEntry
-{
- Oid t_id;
- PgStat_TableCounts t_counts;
-} PgStat_TableEntry;
-
-/* ----------
- * PgStat_MsgTabstat Sent by the backend to report table
- * and buffer access statistics.
- * ----------
- */
-#define PGSTAT_NUM_TABENTRIES \
- ((PGSTAT_MSG_PAYLOAD - sizeof(Oid) - 3 * sizeof(int) - 2 * sizeof(PgStat_Counter)) \
- / sizeof(PgStat_TableEntry))
-
-typedef struct PgStat_MsgTabstat
-{
- PgStat_MsgHdr m_hdr;
- Oid m_databaseid;
- int m_nentries;
- int m_xact_commit;
- int m_xact_rollback;
- PgStat_Counter m_block_read_time; /* times in microseconds */
- PgStat_Counter m_block_write_time;
- PgStat_TableEntry m_entry[PGSTAT_NUM_TABENTRIES];
-} PgStat_MsgTabstat;
-
-
-/* ----------
- * PgStat_MsgTabpurge Sent by the backend to tell the collector
- * about dead tables.
- * ----------
- */
-#define PGSTAT_NUM_TABPURGE \
- ((PGSTAT_MSG_PAYLOAD - sizeof(Oid) - sizeof(int)) \
- / sizeof(Oid))
-
-typedef struct PgStat_MsgTabpurge
-{
- PgStat_MsgHdr m_hdr;
- Oid m_databaseid;
- int m_nentries;
- Oid m_tableid[PGSTAT_NUM_TABPURGE];
-} PgStat_MsgTabpurge;
-
-
-/* ----------
- * PgStat_MsgDropdb Sent by the backend to tell the collector
- * about a dropped database
- * ----------
- */
-typedef struct PgStat_MsgDropdb
-{
- PgStat_MsgHdr m_hdr;
- Oid m_databaseid;
-} PgStat_MsgDropdb;
-
-
-/* ----------
- * PgStat_MsgResetcounter Sent by the backend to tell the collector
- * to reset counters
- * ----------
- */
-typedef struct PgStat_MsgResetcounter
-{
- PgStat_MsgHdr m_hdr;
- Oid m_databaseid;
-} PgStat_MsgResetcounter;
-
-/* ----------
- * PgStat_MsgResetsharedcounter Sent by the backend to tell the collector
- * to reset a shared counter
- * ----------
- */
-typedef struct PgStat_MsgResetsharedcounter
-{
- PgStat_MsgHdr m_hdr;
- PgStat_Shared_Reset_Target m_resettarget;
-} PgStat_MsgResetsharedcounter;
-
-/* ----------
- * PgStat_MsgResetsinglecounter Sent by the backend to tell the collector
- * to reset a single counter
- * ----------
- */
-typedef struct PgStat_MsgResetsinglecounter
-{
- PgStat_MsgHdr m_hdr;
- Oid m_databaseid;
- PgStat_Single_Reset_Type m_resettype;
- Oid m_objectid;
-} PgStat_MsgResetsinglecounter;
-
-/* ----------
- * PgStat_MsgResetslrucounter Sent by the backend to tell the collector
- * to reset a SLRU counter
- * ----------
- */
-typedef struct PgStat_MsgResetslrucounter
-{
- PgStat_MsgHdr m_hdr;
- int m_index;
-} PgStat_MsgResetslrucounter;
-
-/* ----------
- * PgStat_MsgResetreplslotcounter Sent by the backend to tell the collector
- * to reset replication slot counter(s)
- * ----------
- */
-typedef struct PgStat_MsgResetreplslotcounter
-{
- PgStat_MsgHdr m_hdr;
- char m_slotname[NAMEDATALEN];
- bool clearall;
-} PgStat_MsgResetreplslotcounter;
-
-/* ----------
- * PgStat_MsgAutovacStart Sent by the autovacuum daemon to signal
- * that a database is going to be processed
- * ----------
- */
-typedef struct PgStat_MsgAutovacStart
-{
- PgStat_MsgHdr m_hdr;
- Oid m_databaseid;
- TimestampTz m_start_time;
-} PgStat_MsgAutovacStart;
-
-
-/* ----------
- * PgStat_MsgVacuum Sent by the backend or autovacuum daemon
- * after VACUUM
- * ----------
- */
-typedef struct PgStat_MsgVacuum
-{
- PgStat_MsgHdr m_hdr;
- Oid m_databaseid;
- Oid m_tableoid;
- bool m_autovacuum;
- TimestampTz m_vacuumtime;
- PgStat_Counter m_live_tuples;
- PgStat_Counter m_dead_tuples;
-} PgStat_MsgVacuum;
-
-
-/* ----------
- * PgStat_MsgAnalyze Sent by the backend or autovacuum daemon
- * after ANALYZE
- * ----------
- */
-typedef struct PgStat_MsgAnalyze
-{
- PgStat_MsgHdr m_hdr;
- Oid m_databaseid;
- Oid m_tableoid;
- bool m_autovacuum;
- bool m_resetcounter;
- TimestampTz m_analyzetime;
- PgStat_Counter m_live_tuples;
- PgStat_Counter m_dead_tuples;
-} PgStat_MsgAnalyze;
-
-
-/* ----------
- * PgStat_MsgArchiver Sent by the archiver to update statistics.
- * ----------
- */
-typedef struct PgStat_MsgArchiver
-{
- PgStat_MsgHdr m_hdr;
- bool m_failed; /* Failed attempt */
- char m_xlog[MAX_XFN_CHARS + 1];
- TimestampTz m_timestamp;
-} PgStat_MsgArchiver;
-
-/* ----------
- * PgStat_MsgBgWriter Sent by the bgwriter to update statistics.
- * ----------
- */
-typedef struct PgStat_MsgBgWriter
-{
- PgStat_MsgHdr m_hdr;
-
- PgStat_Counter m_timed_checkpoints;
- PgStat_Counter m_requested_checkpoints;
- PgStat_Counter m_buf_written_checkpoints;
- PgStat_Counter m_buf_written_clean;
- PgStat_Counter m_maxwritten_clean;
- PgStat_Counter m_buf_written_backend;
- PgStat_Counter m_buf_fsync_backend;
- PgStat_Counter m_buf_alloc;
- PgStat_Counter m_checkpoint_write_time; /* times in milliseconds */
- PgStat_Counter m_checkpoint_sync_time;
-} PgStat_MsgBgWriter;
-
-/* ----------
- * PgStat_MsgWal Sent by backends and background processes to update WAL statistics.
- * ----------
- */
-typedef struct PgStat_MsgWal
-{
- PgStat_MsgHdr m_hdr;
- PgStat_Counter m_wal_records;
- PgStat_Counter m_wal_fpi;
- uint64 m_wal_bytes;
- PgStat_Counter m_wal_buffers_full;
- PgStat_Counter m_wal_write;
- PgStat_Counter m_wal_sync;
- PgStat_Counter m_wal_write_time; /* time spent writing wal records in
- * microseconds */
- PgStat_Counter m_wal_sync_time; /* time spent syncing wal records in
- * microseconds */
-} PgStat_MsgWal;
-
-/* ----------
- * PgStat_MsgSLRU Sent by a backend to update SLRU statistics.
- * ----------
- */
-typedef struct PgStat_MsgSLRU
-{
- PgStat_MsgHdr m_hdr;
- PgStat_Counter m_index;
- PgStat_Counter m_blocks_zeroed;
- PgStat_Counter m_blocks_hit;
- PgStat_Counter m_blocks_read;
- PgStat_Counter m_blocks_written;
- PgStat_Counter m_blocks_exists;
- PgStat_Counter m_flush;
- PgStat_Counter m_truncate;
-} PgStat_MsgSLRU;
-
-/* ----------
- * PgStat_MsgReplSlot Sent by a backend or a wal sender to update replication
- * slot statistics.
- * ----------
- */
-typedef struct PgStat_MsgReplSlot
-{
- PgStat_MsgHdr m_hdr;
- char m_slotname[NAMEDATALEN];
- bool m_drop;
- PgStat_Counter m_spill_txns;
- PgStat_Counter m_spill_count;
- PgStat_Counter m_spill_bytes;
- PgStat_Counter m_stream_txns;
- PgStat_Counter m_stream_count;
- PgStat_Counter m_stream_bytes;
-} PgStat_MsgReplSlot;
-
-
-/* ----------
- * PgStat_MsgRecoveryConflict Sent by the backend upon recovery conflict
- * ----------
- */
-typedef struct PgStat_MsgRecoveryConflict
-{
- PgStat_MsgHdr m_hdr;
-
- Oid m_databaseid;
- int m_reason;
-} PgStat_MsgRecoveryConflict;
-
-/* ----------
- * PgStat_MsgTempFile Sent by the backend upon creating a temp file
- * ----------
- */
-typedef struct PgStat_MsgTempFile
-{
- PgStat_MsgHdr m_hdr;
-
- Oid m_databaseid;
- size_t m_filesize;
-} PgStat_MsgTempFile;
+ PgStat_Counter timed_checkpoints;
+ PgStat_Counter requested_checkpoints;
+ PgStat_Counter buf_written_checkpoints;
+ PgStat_Counter buf_written_backend;
+ PgStat_Counter buf_fsync_backend;
+ PgStat_Counter checkpoint_write_time; /* times in milliseconds */
+ PgStat_Counter checkpoint_sync_time;
+} PgStat_CheckPointer;
/* ----------
* PgStat_FunctionCounts The actual per-function counts kept by a backend
*
* This struct should contain only actual event counters, because we memcmp
- * it against zeroes to detect whether there are any counts to transmit.
+ * it against zeroes to detect whether there are any counts to write.
*
* Note that the time counters are in instr_time format here. We convert to
- * microseconds in PgStat_Counter format when transmitting to the collector.
+ * microseconds in PgStat_Counter format when writing to shared statsitics.
* ----------
*/
typedef struct PgStat_FunctionCounts
@@ -565,7 +235,6 @@ typedef struct PgStat_FunctionCounts
*/
typedef struct PgStat_BackendFunctionEntry
{
- Oid f_id;
PgStat_FunctionCounts f_counts;
} PgStat_BackendFunctionEntry;
@@ -581,117 +250,8 @@ typedef struct PgStat_FunctionEntry
PgStat_Counter f_self_time;
} PgStat_FunctionEntry;
-/* ----------
- * PgStat_MsgFuncstat Sent by the backend to report function
- * usage statistics.
- * ----------
- */
-#define PGSTAT_NUM_FUNCENTRIES \
- ((PGSTAT_MSG_PAYLOAD - sizeof(Oid) - sizeof(int)) \
- / sizeof(PgStat_FunctionEntry))
-
-typedef struct PgStat_MsgFuncstat
-{
- PgStat_MsgHdr m_hdr;
- Oid m_databaseid;
- int m_nentries;
- PgStat_FunctionEntry m_entry[PGSTAT_NUM_FUNCENTRIES];
-} PgStat_MsgFuncstat;
-
-/* ----------
- * PgStat_MsgFuncpurge Sent by the backend to tell the collector
- * about dead functions.
- * ----------
- */
-#define PGSTAT_NUM_FUNCPURGE \
- ((PGSTAT_MSG_PAYLOAD - sizeof(Oid) - sizeof(int)) \
- / sizeof(Oid))
-
-typedef struct PgStat_MsgFuncpurge
-{
- PgStat_MsgHdr m_hdr;
- Oid m_databaseid;
- int m_nentries;
- Oid m_functionid[PGSTAT_NUM_FUNCPURGE];
-} PgStat_MsgFuncpurge;
-
-/* ----------
- * PgStat_MsgDeadlock Sent by the backend to tell the collector
- * about a deadlock that occurred.
- * ----------
- */
-typedef struct PgStat_MsgDeadlock
-{
- PgStat_MsgHdr m_hdr;
- Oid m_databaseid;
-} PgStat_MsgDeadlock;
-
-/* ----------
- * PgStat_MsgChecksumFailure Sent by the backend to tell the collector
- * about checksum failures noticed.
- * ----------
- */
-typedef struct PgStat_MsgChecksumFailure
-{
- PgStat_MsgHdr m_hdr;
- Oid m_databaseid;
- int m_failurecount;
- TimestampTz m_failure_time;
-} PgStat_MsgChecksumFailure;
-
-/* ----------
- * PgStat_MsgConn Sent by the backend to update connection statistics.
- * ----------
- */
-typedef struct PgStat_MsgConn
-{
- PgStat_MsgHdr m_hdr;
- Oid m_databaseid;
- PgStat_Counter m_count;
- PgStat_Counter m_session_time;
- PgStat_Counter m_active_time;
- PgStat_Counter m_idle_in_xact_time;
- SessionEndType m_disconnect;
-} PgStat_MsgConn;
-
-
-/* ----------
- * PgStat_Msg Union over all possible messages.
- * ----------
- */
-typedef union PgStat_Msg
-{
- PgStat_MsgHdr msg_hdr;
- PgStat_MsgDummy msg_dummy;
- PgStat_MsgInquiry msg_inquiry;
- PgStat_MsgTabstat msg_tabstat;
- PgStat_MsgTabpurge msg_tabpurge;
- PgStat_MsgDropdb msg_dropdb;
- PgStat_MsgResetcounter msg_resetcounter;
- PgStat_MsgResetsharedcounter msg_resetsharedcounter;
- PgStat_MsgResetsinglecounter msg_resetsinglecounter;
- PgStat_MsgResetslrucounter msg_resetslrucounter;
- PgStat_MsgResetreplslotcounter msg_resetreplslotcounter;
- PgStat_MsgAutovacStart msg_autovacuum_start;
- PgStat_MsgVacuum msg_vacuum;
- PgStat_MsgAnalyze msg_analyze;
- PgStat_MsgArchiver msg_archiver;
- PgStat_MsgBgWriter msg_bgwriter;
- PgStat_MsgWal msg_wal;
- PgStat_MsgSLRU msg_slru;
- PgStat_MsgFuncstat msg_funcstat;
- PgStat_MsgFuncpurge msg_funcpurge;
- PgStat_MsgRecoveryConflict msg_recoveryconflict;
- PgStat_MsgDeadlock msg_deadlock;
- PgStat_MsgTempFile msg_tempfile;
- PgStat_MsgChecksumFailure msg_checksumfailure;
- PgStat_MsgReplSlot msg_replslot;
- PgStat_MsgConn msg_conn;
-} PgStat_Msg;
-
-
/* ------------------------------------------------------------
- * Statistic collector data structures follow
+ * Activity statistics data structures on file and shared memory follow
*
* PGSTAT_FILE_FORMAT_ID should be changed whenever any of these
* data structures change.
@@ -700,13 +260,9 @@ typedef union PgStat_Msg
#define PGSTAT_FILE_FORMAT_ID 0x01A5BCA1
-/* ----------
- * PgStat_StatDBEntry The collector's data per database
- * ----------
- */
-typedef struct PgStat_StatDBEntry
+
+typedef struct PgStat_StatDBCounts
{
- Oid databaseid;
PgStat_Counter n_xact_commit;
PgStat_Counter n_xact_rollback;
PgStat_Counter n_blocks_fetched;
@@ -716,7 +272,6 @@ typedef struct PgStat_StatDBEntry
PgStat_Counter n_tuples_inserted;
PgStat_Counter n_tuples_updated;
PgStat_Counter n_tuples_deleted;
- TimestampTz last_autovac_time;
PgStat_Counter n_conflict_tablespace;
PgStat_Counter n_conflict_lock;
PgStat_Counter n_conflict_snapshot;
@@ -726,7 +281,6 @@ typedef struct PgStat_StatDBEntry
PgStat_Counter n_temp_bytes;
PgStat_Counter n_deadlocks;
PgStat_Counter n_checksum_failures;
- TimestampTz last_checksum_failure;
PgStat_Counter n_block_read_time; /* times in microseconds */
PgStat_Counter n_block_write_time;
PgStat_Counter n_sessions;
@@ -736,26 +290,88 @@ typedef struct PgStat_StatDBEntry
PgStat_Counter n_sessions_abandoned;
PgStat_Counter n_sessions_fatal;
PgStat_Counter n_sessions_killed;
+} PgStat_StatDBCounts;
+/* ----------
+ * PgStat_StatEntryHead common header struct for PgStat_Stat*Entry
+ * ----------
+ */
+typedef struct PgStat_StatEntryHeader
+{
+ LWLock lock;
+ bool dropped; /* This entry is being dropped and should
+ * be removed when refcount goes to
+ * zero. */
+ pg_atomic_uint32 refcount; /* How many backends are referenceing */
+} PgStat_StatEntryHeader;
+
+/* ----------
+ * PgStat_StatDBEntry The statistics per database
+ * ----------
+ */
+typedef struct PgStat_StatDBEntry
+{
+ PgStat_StatEntryHeader header; /* must be the first member,
+ used only on shared memory */
+ TimestampTz last_autovac_time;
+ TimestampTz last_checksum_failure;
TimestampTz stat_reset_timestamp;
- TimestampTz stats_timestamp; /* time of db stats file update */
+ TimestampTz stats_timestamp; /* time of db stats update */
+
+ PgStat_StatDBCounts counts;
/*
- * tables and functions must be last in the struct, because we don't write
- * the pointers out to the stats file.
+ * The followings must be last in the struct, because we don't write them
+ * out to the stats file.
*/
- HTAB *tables;
- HTAB *functions;
+ dshash_table_handle tables; /* current gen tables hash */
+ dshash_table_handle functions; /* current gen functions hash */
} PgStat_StatDBEntry;
+/* ----------
+ * PgStat_Wal Sent by backends and background processes to
+ * update WAL statistics.
+ * ----------
+ */
+typedef struct PgStat_Wal
+{
+ WalUsage wal_usage;
+ PgStat_Counter wal_buffers_full;
+ PgStat_Counter wal_write;
+ PgStat_Counter wal_sync;
+ PgStat_Counter wal_write_time;
+ PgStat_Counter wal_sync_time;
+ TimestampTz stat_reset_timestamp;
+} PgStat_Wal;
+
+/*
+ * SLRU statistics kept in the stats collector
+ */
+typedef struct PgStat_SLRUStats
+{
+ PgStat_Counter blocks_zeroed;
+ PgStat_Counter blocks_hit;
+ PgStat_Counter blocks_read;
+ PgStat_Counter blocks_written;
+ PgStat_Counter blocks_exists;
+ PgStat_Counter flush;
+ PgStat_Counter truncate;
+ TimestampTz stat_reset_timestamp;
+} PgStat_SLRUStats;
/* ----------
- * PgStat_StatTabEntry The collector's data per table (or index)
+ * PgStat_StatTabEntry The statistics per table (or index)
* ----------
*/
typedef struct PgStat_StatTabEntry
{
- Oid tableid;
+ PgStat_StatEntryHeader header; /* must be the first member,
+ used only on shared memory */
+ /* Persistent data follow */
+ TimestampTz vacuum_timestamp; /* user initiated vacuum */
+ TimestampTz autovac_vacuum_timestamp; /* autovacuum initiated */
+ TimestampTz analyze_timestamp; /* user initiated */
+ TimestampTz autovac_analyze_timestamp; /* autovacuum initiated */
PgStat_Counter numscans;
@@ -775,103 +391,35 @@ typedef struct PgStat_StatTabEntry
PgStat_Counter blocks_fetched;
PgStat_Counter blocks_hit;
- TimestampTz vacuum_timestamp; /* user initiated vacuum */
PgStat_Counter vacuum_count;
- TimestampTz autovac_vacuum_timestamp; /* autovacuum initiated */
PgStat_Counter autovac_vacuum_count;
- TimestampTz analyze_timestamp; /* user initiated */
PgStat_Counter analyze_count;
- TimestampTz autovac_analyze_timestamp; /* autovacuum initiated */
PgStat_Counter autovac_analyze_count;
} PgStat_StatTabEntry;
/* ----------
- * PgStat_StatFuncEntry The collector's data per function
+ * PgStat_StatFuncEntry per function stats data
* ----------
*/
typedef struct PgStat_StatFuncEntry
{
- Oid functionid;
-
+ PgStat_StatEntryHeader header; /* must be the first member,
+ used only on shared memory */
+ /* Persistent data follow */
PgStat_Counter f_numcalls;
PgStat_Counter f_total_time; /* times in microseconds */
PgStat_Counter f_self_time;
} PgStat_StatFuncEntry;
-
-/*
- * Archiver statistics kept in the stats collector
- */
-typedef struct PgStat_ArchiverStats
-{
- PgStat_Counter archived_count; /* archival successes */
- char last_archived_wal[MAX_XFN_CHARS + 1]; /* last WAL file
- * archived */
- TimestampTz last_archived_timestamp; /* last archival success time */
- PgStat_Counter failed_count; /* failed archival attempts */
- char last_failed_wal[MAX_XFN_CHARS + 1]; /* WAL file involved in
- * last failure */
- TimestampTz last_failed_timestamp; /* last archival failure time */
- TimestampTz stat_reset_timestamp;
-} PgStat_ArchiverStats;
-
-/*
- * Global statistics kept in the stats collector
- */
-typedef struct PgStat_GlobalStats
-{
- TimestampTz stats_timestamp; /* time of stats file update */
- PgStat_Counter timed_checkpoints;
- PgStat_Counter requested_checkpoints;
- PgStat_Counter checkpoint_write_time; /* times in milliseconds */
- PgStat_Counter checkpoint_sync_time;
- PgStat_Counter buf_written_checkpoints;
- PgStat_Counter buf_written_clean;
- PgStat_Counter maxwritten_clean;
- PgStat_Counter buf_written_backend;
- PgStat_Counter buf_fsync_backend;
- PgStat_Counter buf_alloc;
- TimestampTz stat_reset_timestamp;
-} PgStat_GlobalStats;
-
-/*
- * WAL statistics kept in the stats collector
- */
-typedef struct PgStat_WalStats
-{
- PgStat_Counter wal_records;
- PgStat_Counter wal_fpi;
- uint64 wal_bytes;
- PgStat_Counter wal_buffers_full;
- PgStat_Counter wal_write;
- PgStat_Counter wal_sync;
- PgStat_Counter wal_write_time;
- PgStat_Counter wal_sync_time;
- TimestampTz stat_reset_timestamp;
-} PgStat_WalStats;
-
-/*
- * SLRU statistics kept in the stats collector
- */
-typedef struct PgStat_SLRUStats
-{
- PgStat_Counter blocks_zeroed;
- PgStat_Counter blocks_hit;
- PgStat_Counter blocks_read;
- PgStat_Counter blocks_written;
- PgStat_Counter blocks_exists;
- PgStat_Counter flush;
- PgStat_Counter truncate;
- TimestampTz stat_reset_timestamp;
-} PgStat_SLRUStats;
-
/*
* Replication slot statistics kept in the stats collector
*/
-typedef struct PgStat_ReplSlotStats
+typedef struct PgStat_ReplSlot
{
+ PgStat_StatEntryHeader header; /* must be the first member,
+ used only on shared memory */
char slotname[NAMEDATALEN];
PgStat_Counter spill_txns;
PgStat_Counter spill_count;
@@ -880,7 +428,7 @@ typedef struct PgStat_ReplSlotStats
PgStat_Counter stream_count;
PgStat_Counter stream_bytes;
TimestampTz stat_reset_timestamp;
-} PgStat_ReplSlotStats;
+} PgStat_ReplSlot;
/* ----------
* Backend states
@@ -929,7 +477,7 @@ typedef enum
WAIT_EVENT_CHECKPOINTER_MAIN,
WAIT_EVENT_LOGICAL_APPLY_MAIN,
WAIT_EVENT_LOGICAL_LAUNCHER_MAIN,
- WAIT_EVENT_PGSTAT_MAIN,
+ WAIT_EVENT_READING_STATS_FILE,
WAIT_EVENT_RECOVERY_WAL_STREAM,
WAIT_EVENT_SYSLOGGER_MAIN,
WAIT_EVENT_WAL_RECEIVER_MAIN,
@@ -1181,7 +729,7 @@ typedef struct PgBackendGSSStatus
*
* Each live backend maintains a PgBackendStatus struct in shared memory
* showing its current activity. (The structs are allocated according to
- * BackendId, but that is not critical.) Note that the collector process
+ * BackendId, but that is not critical.) Note that the stats-writing functions
* has no involvement in, or even access to, these structs.
*
* Each auxiliary process also maintains a PgBackendStatus struct in shared
@@ -1378,18 +926,26 @@ extern PGDLLIMPORT bool pgstat_track_counts;
extern PGDLLIMPORT int pgstat_track_functions;
extern PGDLLIMPORT int pgstat_track_activity_query_size;
extern char *pgstat_stat_directory;
+
+/* No longer used, but will be removed with GUC */
extern char *pgstat_stat_tmpname;
extern char *pgstat_stat_filename;
/*
* BgWriter statistics counters are updated directly by bgwriter and bufmgr
*/
-extern PgStat_MsgBgWriter BgWriterStats;
+extern PgStat_BgWriter BgWriterStats;
+
+/*
+ * CheckPointer statistics counters are updated directly by checkpointer and
+ * bufmgr
+ */
+extern PgStat_CheckPointer CheckPointerStats;
/*
* WAL statistics counter is updated by backends and background processes
*/
-extern PgStat_MsgWal WalStats;
+extern PgStat_Wal WalStats;
/*
* Updated by pgstat_count_buffer_*_time macros
@@ -1409,33 +965,27 @@ extern SessionEndType pgStatSessionEndCause;
extern Size BackendStatusShmemSize(void);
extern void CreateSharedBackendStatus(void);
-extern void pgstat_init(void);
-extern int pgstat_start(void);
+extern Size StatsShmemSize(void);
+extern void StatsShmemInit(void);
+
extern void pgstat_reset_all(void);
-extern void allow_immediate_pgstat_restart(void);
-
-#ifdef EXEC_BACKEND
-extern void PgstatCollectorMain(int argc, char *argv[]) pg_attribute_noreturn();
-#endif
-
/* ----------
* Functions called from backends
* ----------
*/
-extern void pgstat_ping(void);
-
-extern void pgstat_report_stat(bool force);
+extern long pgstat_report_stat(bool force);
extern void pgstat_vacuum_stat(void);
extern void pgstat_drop_database(Oid databaseid);
extern void pgstat_clear_snapshot(void);
extern void pgstat_reset_counters(void);
-extern void pgstat_reset_shared_counters(const char *);
+extern void pgstat_reset_shared_counters(const char *target);
extern void pgstat_reset_single_counter(Oid objectid, PgStat_Single_Reset_Type type);
extern void pgstat_reset_slru_counter(const char *);
extern void pgstat_reset_replslot_counter(const char *name);
+extern void pgstat_report_wal(void);
extern void pgstat_report_autovac(Oid dboid);
extern void pgstat_report_vacuum(Oid tableoid, bool shared,
PgStat_Counter livetuples, PgStat_Counter deadtuples);
@@ -1475,6 +1025,7 @@ extern PgStat_TableStatus *find_tabstat_entry(Oid rel_id);
extern PgStat_BackendFunctionEntry *find_funcstat_entry(Oid func_id);
extern void pgstat_initstats(Relation rel);
+extern void pgstat_delinkstats(Relation rel);
extern char *pgstat_clip_activity(const char *raw_activity);
@@ -1597,10 +1148,9 @@ extern void pgstat_twophase_postcommit(TransactionId xid, uint16 info,
extern void pgstat_twophase_postabort(TransactionId xid, uint16 info,
void *recdata, uint32 len);
-extern void pgstat_send_archiver(const char *xlog, bool failed);
-extern void pgstat_send_bgwriter(void);
-extern void pgstat_report_wal(void);
-extern bool pgstat_send_wal(bool force);
+extern void pgstat_report_archiver(const char *xlog, bool failed);
+extern void pgstat_report_bgwriter(void);
+extern void pgstat_report_checkpointer(void);
/* ----------
* Support functions for the SQL-callable functions to
@@ -1609,15 +1159,20 @@ extern bool pgstat_send_wal(bool force);
*/
extern PgStat_StatDBEntry *pgstat_fetch_stat_dbentry(Oid dbid);
extern PgStat_StatTabEntry *pgstat_fetch_stat_tabentry(Oid relid);
+extern PgStat_StatTabEntry *pgstat_fetch_stat_tabentry_extended(bool shared,
+ Oid relid);
+extern void pgstat_copy_index_counters(Oid relid, PgStat_TableStatus *dst);
extern PgBackendStatus *pgstat_fetch_stat_beentry(int beid);
extern LocalPgBackendStatus *pgstat_fetch_stat_local_beentry(int beid);
extern PgStat_StatFuncEntry *pgstat_fetch_stat_funcentry(Oid funcid);
extern int pgstat_fetch_stat_numbackends(void);
-extern PgStat_ArchiverStats *pgstat_fetch_stat_archiver(void);
-extern PgStat_GlobalStats *pgstat_fetch_global(void);
-extern PgStat_WalStats *pgstat_fetch_stat_wal(void);
+extern TimestampTz pgstat_get_stat_timestamp(void);
+extern PgStat_Archiver *pgstat_fetch_stat_archiver(void);
+extern PgStat_BgWriter *pgstat_fetch_stat_bgwriter(void);
+extern PgStat_CheckPointer *pgstat_fetch_stat_checkpointer(void);
+extern PgStat_Wal *pgstat_fetch_stat_wal(void);
extern PgStat_SLRUStats *pgstat_fetch_slru(void);
-extern PgStat_ReplSlotStats *pgstat_fetch_replslot(int *nslots_p);
+extern PgStat_ReplSlot *pgstat_fetch_replslot(int *nslots_p);
extern void pgstat_count_slru_page_zeroed(int slru_idx);
extern void pgstat_count_slru_page_hit(int slru_idx);
@@ -1628,5 +1183,6 @@ extern void pgstat_count_slru_flush(int slru_idx);
extern void pgstat_count_slru_truncate(int slru_idx);
extern const char *pgstat_slru_name(int slru_idx);
extern int pgstat_slru_index(const char *name);
+extern void pgstat_clear_snapshot(void);
#endif /* PGSTAT_H */
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index cbf2510fbf..9ed6b54428 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -218,6 +218,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_SHARED_TIDBITMAP,
LWTRANCHE_PARALLEL_APPEND,
LWTRANCHE_PER_XACT_PREDICATE_LIST,
+ LWTRANCHE_STATS,
LWTRANCHE_FIRST_USER_DEFINED
} BuiltinTrancheIds;
diff --git a/src/include/utils/guc_tables.h b/src/include/utils/guc_tables.h
index b9b5c1adda..add9c53ee3 100644
--- a/src/include/utils/guc_tables.h
+++ b/src/include/utils/guc_tables.h
@@ -88,7 +88,7 @@ enum config_group
PROCESS_TITLE,
STATS,
STATS_MONITORING,
- STATS_COLLECTOR,
+ STATS_ACTIVITY,
AUTOVACUUM,
CLIENT_CONN,
CLIENT_CONN_STATEMENT,
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index ecb2a366a5..f090f7372a 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -32,6 +32,7 @@ typedef enum TimeoutId
STANDBY_LOCK_TIMEOUT,
IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
IDLE_SESSION_TIMEOUT,
+ IDLE_STATS_UPDATE_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
/* Maximum number of timeout reasons */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 2b591e94e6..f96dd10924 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1830,52 +1830,33 @@ PgFdwPathExtraData
PgFdwRelationInfo
PgFdwScanState
PgIfAddrCallback
-PgStat_ArchiverStats
+PgStat_Archiver
+PgStat_BgWriter
+PgStat_CheckPointer
+PgStat_ReplSlot
+PgStat_SLRUStats
+PgStat_StatDBCounts
+PgStat_StatDBEntry
+PgStat_StatEntryHeader
+PgStat_Wal
PgStat_BackendFunctionEntry
PgStat_Counter
PgStat_FunctionCallUsage
PgStat_FunctionCounts
PgStat_FunctionEntry
-PgStat_GlobalStats
-PgStat_Msg
-PgStat_MsgAnalyze
-PgStat_MsgArchiver
-PgStat_MsgAutovacStart
-PgStat_MsgBgWriter
-PgStat_MsgChecksumFailure
-PgStat_MsgDeadlock
-PgStat_MsgDropdb
-PgStat_MsgDummy
-PgStat_MsgFuncpurge
-PgStat_MsgFuncstat
-PgStat_MsgHdr
-PgStat_MsgInquiry
-PgStat_MsgRecoveryConflict
-PgStat_MsgReplSlot
-PgStat_MsgResetcounter
-PgStat_MsgResetreplslotcounter
-PgStat_MsgResetsharedcounter
-PgStat_MsgResetsinglecounter
-PgStat_MsgResetslrucounter
-PgStat_MsgSLRU
-PgStat_MsgTabpurge
-PgStat_MsgTabstat
-PgStat_MsgTempFile
-PgStat_MsgVacuum
-PgStat_MsgWal
-PgStat_ReplSlotStats
-PgStat_SLRUStats
PgStat_Shared_Reset_Target
PgStat_Single_Reset_Type
-PgStat_StatDBEntry
PgStat_StatFuncEntry
PgStat_StatTabEntry
PgStat_SubXactStatus
PgStat_TableCounts
-PgStat_TableEntry
PgStat_TableStatus
PgStat_TableXactStatus
-PgStat_WalStats
+PgStatHashEntry
+PgStatHashKey
+PgStatLocalHashEntry
+PgStatSharedSLRUStats
+PgStatTypes
PgXmlErrorContext
PgXmlStrictness
Pg_finfo_record
@@ -2389,12 +2370,12 @@ StartupPacket
StartupStatusEnum
StatEntry
StatExtEntry
-StatMsgType
StateFileChunk
StatisticExtInfo
Stats
StatsData
StatsExtInfo
+StatsShmemStruct
StdAnalyzeData
StdRdOptions
Step
@@ -2492,8 +2473,6 @@ TXNEntryFile
TYPCATEGORY
T_Action
T_WorkerStatus
-TabStatHashEntry
-TabStatusArray
TableAmRoutine
TableDataInfo
TableFunc
@@ -3249,6 +3228,7 @@ pgssLocationLen
pgssSharedState
pgssStoreKind
pgssVersion
+pgstat_oident
pgstat_page
pgstattuple_type
pgthreadlock_t
--
2.27.0
----Next_Part(Tue_Mar__9_18_29_34_2021_806)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v50-0005-Doc-part-of-shared-memory-based-stats-collector.patch"
^ permalink raw reply [nested|flat] 5+ messages in thread
* RE: Handle infinite recursion in logical replication setup
@ 2022-03-01 10:42 [email protected] <[email protected]>
0 siblings, 2 replies; 5+ messages in thread
From: [email protected] @ 2022-03-01 10:42 UTC (permalink / raw)
To: 'vignesh C' <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Hi Vignesh,
> In logical replication, currently Walsender sends the data that is
> generated locally and the data that are replicated from other
> instances. This results in infinite recursion in circular logical
> replication setup.
Thank you for good explanation. I understand that this fix can be used
for a bidirectional replication.
> Here there are two problems for the user: a) incremental
> synchronization of table sending both local data and replicated data
> by walsender b) Table synchronization of table using copy command
> sending both local data and replicated data
So you wanted to solve these two problem and currently focused on
the first one, right? We can check one by one.
> For the first problem "Incremental synchronization of table by
> Walsender" can be solved by:
> Currently the locally generated data does not have replication origin
> associated and the data that has originated from another instance will
> have a replication origin associated. We could use this information to
> differentiate locally generated data and replicated data and send only
> the locally generated data. This "only_local" could be provided as an
> option while subscription is created:
> ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=5433'
> PUBLICATION pub1 with (only_local = on);
Sounds good, but I cannot distinguish whether the assumption will keep.
I played with your patch, but it could not be applied to current master.
I tested from bd74c40 and I confirmed infinite loop was not appeared.
local_only could not be set from ALTER SUBSCRIPTION command.
Is it expected?
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Handle infinite recursion in logical replication setup
@ 2022-03-02 16:12 vignesh C <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 1 reply; 5+ messages in thread
From: vignesh C @ 2022-03-02 16:12 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Tue, Mar 1, 2022 at 4:12 PM [email protected]
<[email protected]> wrote:
>
> Hi Vignesh,
>
> > In logical replication, currently Walsender sends the data that is
> > generated locally and the data that are replicated from other
> > instances. This results in infinite recursion in circular logical
> > replication setup.
>
> Thank you for good explanation. I understand that this fix can be used
> for a bidirectional replication.
>
> > Here there are two problems for the user: a) incremental
> > synchronization of table sending both local data and replicated data
> > by walsender b) Table synchronization of table using copy command
> > sending both local data and replicated data
>
> So you wanted to solve these two problem and currently focused on
> the first one, right? We can check one by one.
>
> > For the first problem "Incremental synchronization of table by
> > Walsender" can be solved by:
> > Currently the locally generated data does not have replication origin
> > associated and the data that has originated from another instance will
> > have a replication origin associated. We could use this information to
> > differentiate locally generated data and replicated data and send only
> > the locally generated data. This "only_local" could be provided as an
> > option while subscription is created:
> > ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=5433'
> > PUBLICATION pub1 with (only_local = on);
>
> Sounds good, but I cannot distinguish whether the assumption will keep.
>
> I played with your patch, but it could not be applied to current master.
> I tested from bd74c40 and I confirmed infinite loop was not appeared.
Rebased the patch on top of head
> local_only could not be set from ALTER SUBSCRIPTION command.
> Is it expected?
Modified
Thanks for the comments, the attached patch has the changes for the same.
Regards,
Vignesh
Attachments:
[text/x-patch] v2-0001-Skip-replication-of-non-local-data.patch (42.8K, ../../CALDaNm0WSo5369pr2eN1obTGBeiJU9cQdF6Ju1sC4hMQNy5BfQ@mail.gmail.com/2-v2-0001-Skip-replication-of-non-local-data.patch)
download | inline diff:
From 7c67cc23584e1106fbf2011c8c6658442125e48f Mon Sep 17 00:00:00 2001
From: Vigneshwaran C <[email protected]>
Date: Wed, 2 Mar 2022 20:40:34 +0530
Subject: [PATCH v2] Skip replication of non local data.
Add an option only_local which will subscribe only to the locally
generated data in the publisher node. If subscriber is created with this
option, publisher will skip publishing the data that was subscribed
from other nodes. It can be created using following syntax:
ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=9999' PUBLICATION pub1 with (only_local = on);
---
contrib/test_decoding/test_decoding.c | 13 +++
doc/src/sgml/ref/alter_subscription.sgml | 3 +-
doc/src/sgml/ref/create_subscription.sgml | 12 ++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 29 ++++-
.../libpqwalreceiver/libpqwalreceiver.c | 18 ++-
src/backend/replication/logical/decode.c | 36 ++++--
src/backend/replication/logical/logical.c | 35 ++++++
src/backend/replication/logical/tablesync.c | 2 +-
src/backend/replication/logical/worker.c | 2 +
src/backend/replication/pgoutput/pgoutput.c | 25 ++++
src/backend/replication/slot.c | 4 +-
src/backend/replication/slotfuncs.c | 18 ++-
src/backend/replication/walreceiver.c | 2 +-
src/backend/replication/walsender.c | 21 +++-
src/bin/psql/tab-complete.c | 4 +-
src/include/catalog/pg_proc.dat | 6 +-
src/include/catalog/pg_subscription.h | 3 +
src/include/replication/logical.h | 4 +
src/include/replication/output_plugin.h | 7 ++
src/include/replication/pgoutput.h | 1 +
src/include/replication/slot.h | 5 +-
src/include/replication/walreceiver.h | 8 +-
src/test/regress/expected/rules.out | 5 +-
src/test/regress/expected/subscription.out | 4 +
src/test/regress/sql/subscription.sql | 4 +
src/test/subscription/t/029_circular.pl | 108 ++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
29 files changed, 345 insertions(+), 39 deletions(-)
create mode 100644 src/test/subscription/t/029_circular.pl
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index ea22649e41..58bc5dbc1c 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -73,6 +73,8 @@ static void pg_decode_truncate(LogicalDecodingContext *ctx,
ReorderBufferChange *change);
static bool pg_decode_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static bool pg_decode_filter_remotedata(LogicalDecodingContext *ctx,
+ RepOriginId origin_id);
static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
@@ -148,6 +150,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->truncate_cb = pg_decode_truncate;
cb->commit_cb = pg_decode_commit_txn;
cb->filter_by_origin_cb = pg_decode_filter;
+ cb->filter_remotedata_cb = pg_decode_filter_remotedata;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
cb->sequence_cb = pg_decode_sequence;
@@ -484,6 +487,16 @@ pg_decode_filter(LogicalDecodingContext *ctx,
return false;
}
+static bool
+pg_decode_filter_remotedata(LogicalDecodingContext *ctx,
+ RepOriginId origin_id)
+{
+ TestDecodingData *data = ctx->output_plugin_private;
+
+ if (data->only_local && origin_id != InvalidRepOriginId)
+ return true;
+ return false;
+}
/*
* Print literal `outputstr' already represented as string of type `typid'
* into stringbuf `s'.
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 0d6f064f58..bd2ef19cce 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -204,7 +204,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
information. The parameters that can be altered
are <literal>slot_name</literal>,
<literal>synchronous_commit</literal>,
- <literal>binary</literal>, and
+ <literal>binary</literal>,
+ <literal>only_local</literal>, and
<literal>streaming</literal>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index e80a2617a3..b5552dc58c 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -152,6 +152,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>only_local</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription should subscribe only to the
+ locally generated changes or subscribe to both the locally generated
+ changes and the replicated data present from the publisher. The
+ default is <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index ca65a8bd20..94e096e5fb 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -69,6 +69,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->binary = subform->subbinary;
sub->stream = subform->substream;
sub->twophasestate = subform->subtwophasestate;
+ sub->onlylocaldata = subform->subonlylocaldata;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 40b7bca5a9..c7653298c5 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -958,7 +958,8 @@ CREATE VIEW pg_replication_slots AS
L.confirmed_flush_lsn,
L.wal_status,
L.safe_wal_size,
- L.two_phase
+ L.two_phase,
+ L.only_local
FROM pg_get_replication_slots() AS L
LEFT JOIN pg_database D ON (L.datoid = D.oid);
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 3ef6607d24..5be0211ac3 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -61,6 +61,7 @@
#define SUBOPT_BINARY 0x00000080
#define SUBOPT_STREAMING 0x00000100
#define SUBOPT_TWOPHASE_COMMIT 0x00000200
+#define SUBOPT_ONLYLOCAL_DATA 0x00000400
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -82,6 +83,7 @@ typedef struct SubOpts
bool binary;
bool streaming;
bool twophase;
+ bool onlylocal_data;
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
@@ -130,6 +132,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->streaming = false;
if (IsSet(supported_opts, SUBOPT_TWOPHASE_COMMIT))
opts->twophase = false;
+ if (IsSet(supported_opts, SUBOPT_ONLYLOCAL_DATA))
+ opts->onlylocal_data = false;
/* Parse options */
foreach(lc, stmt_options)
@@ -228,6 +232,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_STREAMING;
opts->streaming = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_ONLYLOCAL_DATA) &&
+ strcmp(defel->defname, "only_local") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_ONLYLOCAL_DATA))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_ONLYLOCAL_DATA;
+ opts->onlylocal_data = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "two_phase") == 0)
{
/*
@@ -390,7 +403,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
supported_opts = (SUBOPT_CONNECT | SUBOPT_ENABLED | SUBOPT_CREATE_SLOT |
SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT);
+ SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
+ SUBOPT_ONLYLOCAL_DATA);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -460,6 +474,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(opts.enabled);
values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(opts.binary);
values[Anum_pg_subscription_substream - 1] = BoolGetDatum(opts.streaming);
+ values[Anum_pg_subscription_subonlylocaldata - 1] = BoolGetDatum(opts.onlylocal_data);
values[Anum_pg_subscription_subtwophasestate - 1] =
CharGetDatum(opts.twophase ?
LOGICALREP_TWOPHASE_STATE_PENDING :
@@ -565,7 +580,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
twophase_enabled = true;
walrcv_create_slot(wrconn, opts.slot_name, false, twophase_enabled,
- CRS_NOEXPORT_SNAPSHOT, NULL);
+ CRS_NOEXPORT_SNAPSHOT, NULL,
+ opts.onlylocal_data);
if (twophase_enabled)
UpdateTwoPhaseState(subid, LOGICALREP_TWOPHASE_STATE_ENABLED);
@@ -864,7 +880,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
supported_opts = (SUBOPT_SLOT_NAME |
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
- SUBOPT_STREAMING);
+ SUBOPT_STREAMING | SUBOPT_ONLYLOCAL_DATA);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -913,6 +929,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_substream - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_ONLYLOCAL_DATA))
+ {
+ values[Anum_pg_subscription_subonlylocaldata - 1] =
+ BoolGetDatum(opts.streaming);
+ replaces[Anum_pg_subscription_subonlylocaldata - 1] = true;
+ }
+
update_tuple = true;
break;
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0d89db4e6a..326f60414e 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -75,7 +75,8 @@ static char *libpqrcv_create_slot(WalReceiverConn *conn,
bool temporary,
bool two_phase,
CRSSnapshotAction snapshot_action,
- XLogRecPtr *lsn);
+ XLogRecPtr *lsn,
+ bool onlylocal_data);
static pid_t libpqrcv_get_backend_pid(WalReceiverConn *conn);
static WalRcvExecResult *libpqrcv_exec(WalReceiverConn *conn,
const char *query,
@@ -453,6 +454,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
PQserverVersion(conn->streamConn) >= 150000)
appendStringInfoString(&cmd, ", two_phase 'on'");
+ if (options->proto.logical.onlylocal_data &&
+ PQserverVersion(conn->streamConn) >= 150000)
+ appendStringInfoString(&cmd, ", only_local 'on'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
@@ -869,7 +874,7 @@ libpqrcv_send(WalReceiverConn *conn, const char *buffer, int nbytes)
static char *
libpqrcv_create_slot(WalReceiverConn *conn, const char *slotname,
bool temporary, bool two_phase, CRSSnapshotAction snapshot_action,
- XLogRecPtr *lsn)
+ XLogRecPtr *lsn, bool onlylocal_data)
{
PGresult *res;
StringInfoData cmd;
@@ -899,6 +904,15 @@ libpqrcv_create_slot(WalReceiverConn *conn, const char *slotname,
appendStringInfoChar(&cmd, ' ');
}
+ if (onlylocal_data)
+ {
+ appendStringInfoString(&cmd, "ONLY_LOCAL");
+ if (use_new_options_syntax)
+ appendStringInfoString(&cmd, ", ");
+ else
+ appendStringInfoChar(&cmd, ' ');
+ }
+
if (use_new_options_syntax)
{
switch (snapshot_action)
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 18cf931822..6305b93fc7 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -555,6 +555,15 @@ FilterByOrigin(LogicalDecodingContext *ctx, RepOriginId origin_id)
return filter_by_origin_cb_wrapper(ctx, origin_id);
}
+static inline bool
+FilterRemoteOriginData(LogicalDecodingContext *ctx, RepOriginId origin_id)
+{
+ if (ctx->callbacks.filter_remotedata_cb == NULL)
+ return false;
+
+ return filter_remotedata_cb_wrapper(ctx, origin_id);
+}
+
/*
* Handle rmgr LOGICALMSG_ID records for DecodeRecordIntoReorderBuffer().
*/
@@ -585,7 +594,8 @@ logicalmsg_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
message = (xl_logical_message *) XLogRecGetData(r);
if (message->dbId != ctx->slot->data.database ||
- FilterByOrigin(ctx, origin_id))
+ FilterByOrigin(ctx, origin_id) ||
+ FilterRemoteOriginData(ctx, origin_id))
return;
if (message->transactional &&
@@ -864,7 +874,8 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
return;
/* output plugin doesn't look for this origin, no need to queue */
- if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
+ if (FilterByOrigin(ctx, XLogRecGetOrigin(r)) ||
+ FilterRemoteOriginData(ctx, XLogRecGetOrigin(r)))
return;
change = ReorderBufferGetChange(ctx->reorder);
@@ -914,7 +925,8 @@ DecodeUpdate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
return;
/* output plugin doesn't look for this origin, no need to queue */
- if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
+ if (FilterByOrigin(ctx, XLogRecGetOrigin(r)) ||
+ FilterRemoteOriginData(ctx, XLogRecGetOrigin(r)))
return;
change = ReorderBufferGetChange(ctx->reorder);
@@ -980,7 +992,8 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
return;
/* output plugin doesn't look for this origin, no need to queue */
- if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
+ if (FilterByOrigin(ctx, XLogRecGetOrigin(r)) ||
+ FilterRemoteOriginData(ctx, XLogRecGetOrigin(r)))
return;
change = ReorderBufferGetChange(ctx->reorder);
@@ -1032,7 +1045,8 @@ DecodeTruncate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
return;
/* output plugin doesn't look for this origin, no need to queue */
- if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
+ if (FilterByOrigin(ctx, XLogRecGetOrigin(r)) ||
+ FilterRemoteOriginData(ctx, XLogRecGetOrigin(r)))
return;
change = ReorderBufferGetChange(ctx->reorder);
@@ -1082,7 +1096,8 @@ DecodeMultiInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
return;
/* output plugin doesn't look for this origin, no need to queue */
- if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
+ if (FilterByOrigin(ctx, XLogRecGetOrigin(r)) ||
+ FilterRemoteOriginData(ctx, XLogRecGetOrigin(r)))
return;
/*
@@ -1175,7 +1190,8 @@ DecodeSpecConfirm(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
return;
/* output plugin doesn't look for this origin, no need to queue */
- if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
+ if (FilterByOrigin(ctx, XLogRecGetOrigin(r)) ||
+ FilterRemoteOriginData(ctx, XLogRecGetOrigin(r)))
return;
change = ReorderBufferGetChange(ctx->reorder);
@@ -1250,7 +1266,8 @@ DecodeTXNNeedSkip(LogicalDecodingContext *ctx, XLogRecordBuffer *buf,
{
return (SnapBuildXactNeedsSkip(ctx->snapshot_builder, buf->origptr) ||
(txn_dbid != InvalidOid && txn_dbid != ctx->slot->data.database) ||
- ctx->fast_forward || FilterByOrigin(ctx, origin_id));
+ ctx->fast_forward || FilterByOrigin(ctx, origin_id) ||
+ FilterRemoteOriginData(ctx, origin_id));
}
/*
@@ -1335,7 +1352,8 @@ sequence_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
return;
/* output plugin doesn't look for this origin, no need to queue */
- if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
+ if (FilterByOrigin(ctx, XLogRecGetOrigin(r)) ||
+ FilterRemoteOriginData(ctx, XLogRecGetOrigin(r)))
return;
tupledata = XLogRecGetData(r);
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 934aa13f2d..19584eaea7 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -246,6 +246,8 @@ StartupDecodingContext(List *output_plugin_options,
(ctx->callbacks.stream_sequence_cb != NULL) ||
(ctx->callbacks.stream_truncate_cb != NULL);
+ ctx->onlylocal_data = ctx->callbacks.filter_remotedata_cb != NULL;
+
/*
* streaming callbacks
*
@@ -451,6 +453,8 @@ CreateInitDecodingContext(const char *plugin,
*/
ctx->twophase &= slot->data.two_phase;
+ ctx->onlylocal_data &= slot->data.onlylocal_data;
+
ctx->reorder->output_rewrites = ctx->options.receive_rewrites;
return ctx;
@@ -1178,6 +1182,37 @@ filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, RepOriginId origin_id)
return ret;
}
+bool
+filter_remotedata_cb_wrapper(LogicalDecodingContext *ctx,
+ RepOriginId origin_id)
+{
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+ bool ret;
+
+ Assert(!ctx->fast_forward);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "filter_remoteorigin";
+ state.report_location = InvalidXLogRecPtr;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = false;
+
+ /* do the actual work: call callback */
+ ret = ctx->callbacks.filter_remotedata_cb(ctx, origin_id);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+
+ return ret;
+}
+
static void
message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 1659964571..f5093ce8c9 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -1224,7 +1224,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos)
HOLD_INTERRUPTS();
walrcv_create_slot(LogRepWorkerWalRcvConn,
slotname, false /* permanent */ , false /* two_phase */ ,
- CRS_USE_SNAPSHOT, origin_startpos);
+ CRS_USE_SNAPSHOT, origin_startpos, false /* only_local */);
RESUME_INTERRUPTS();
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 7e267f7960..a13b8007e7 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2960,6 +2960,7 @@ maybe_reread_subscription(void)
newsub->binary != MySubscription->binary ||
newsub->stream != MySubscription->stream ||
newsub->owner != MySubscription->owner ||
+ newsub->onlylocaldata != MySubscription->onlylocaldata ||
!equal(newsub->publications, MySubscription->publications))
{
ereport(LOG,
@@ -3569,6 +3570,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.binary = MySubscription->binary;
options.proto.logical.streaming = MySubscription->stream;
options.proto.logical.twophase = false;
+ options.proto.logical.onlylocal_data = MySubscription->onlylocaldata;
if (!am_tablesync_worker())
{
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a0477f..0c9b60bd65 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -55,6 +55,8 @@ static void pgoutput_message(LogicalDecodingContext *ctx,
Size sz, const char *message);
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static bool pgoutput_remoteorigin_filter(LogicalDecodingContext *ctx,
+ RepOriginId origin_id);
static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn);
static void pgoutput_prepare_txn(LogicalDecodingContext *ctx,
@@ -215,6 +217,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_prepared_cb = pgoutput_commit_prepared_txn;
cb->rollback_prepared_cb = pgoutput_rollback_prepared_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
+ cb->filter_remotedata_cb = pgoutput_remoteorigin_filter;
cb->shutdown_cb = pgoutput_shutdown;
/* transaction streaming */
@@ -239,11 +242,13 @@ parse_output_parameters(List *options, PGOutputData *data)
bool messages_option_given = false;
bool streaming_given = false;
bool two_phase_option_given = false;
+ bool onlylocal_data_given = false;
data->binary = false;
data->streaming = false;
data->messages = false;
data->two_phase = false;
+ data->onlylocal_data = false;
foreach(lc, options)
{
@@ -332,6 +337,16 @@ parse_output_parameters(List *options, PGOutputData *data)
data->two_phase = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "only_local") == 0)
+ {
+ if (onlylocal_data_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ onlylocal_data_given = true;
+
+ data->onlylocal_data = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -1450,6 +1465,16 @@ pgoutput_origin_filter(LogicalDecodingContext *ctx,
return false;
}
+static bool
+pgoutput_remoteorigin_filter(LogicalDecodingContext *ctx,
+ RepOriginId origin_id)
+{
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+
+ if (data->onlylocal_data && origin_id != InvalidRepOriginId)
+ return true;
+ return false;
+}
/*
* Shutdown the output plugin.
*
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index caa6b29756..fed01829f3 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -253,7 +253,8 @@ ReplicationSlotValidateName(const char *name, int elevel)
*/
void
ReplicationSlotCreate(const char *name, bool db_specific,
- ReplicationSlotPersistency persistency, bool two_phase)
+ ReplicationSlotPersistency persistency, bool two_phase,
+ bool onlylocal_data)
{
ReplicationSlot *slot = NULL;
int i;
@@ -313,6 +314,7 @@ ReplicationSlotCreate(const char *name, bool db_specific,
slot->data.persistency = persistency;
slot->data.two_phase = two_phase;
slot->data.two_phase_at = InvalidXLogRecPtr;
+ slot->data.onlylocal_data = onlylocal_data;
/* and then data only present in shared memory */
slot->just_dirtied = false;
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 886899afd2..0e0bc1e940 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -42,7 +42,8 @@ create_physical_replication_slot(char *name, bool immediately_reserve,
/* acquire replication slot, this will check for conflicting names */
ReplicationSlotCreate(name, false,
- temporary ? RS_TEMPORARY : RS_PERSISTENT, false);
+ temporary ? RS_TEMPORARY : RS_PERSISTENT,
+ false, false);
if (immediately_reserve)
{
@@ -118,7 +119,8 @@ static void
create_logical_replication_slot(char *name, char *plugin,
bool temporary, bool two_phase,
XLogRecPtr restart_lsn,
- bool find_startpoint)
+ bool find_startpoint,
+ bool onlylocal_data)
{
LogicalDecodingContext *ctx = NULL;
@@ -133,7 +135,8 @@ create_logical_replication_slot(char *name, char *plugin,
* error as well.
*/
ReplicationSlotCreate(name, true,
- temporary ? RS_TEMPORARY : RS_EPHEMERAL, two_phase);
+ temporary ? RS_TEMPORARY : RS_EPHEMERAL, two_phase,
+ onlylocal_data);
/*
* Create logical decoding context to find start point or, if we don't
@@ -171,6 +174,7 @@ pg_create_logical_replication_slot(PG_FUNCTION_ARGS)
Name plugin = PG_GETARG_NAME(1);
bool temporary = PG_GETARG_BOOL(2);
bool two_phase = PG_GETARG_BOOL(3);
+ bool onlylocal_data = PG_GETARG_BOOL(4);
Datum result;
TupleDesc tupdesc;
HeapTuple tuple;
@@ -189,7 +193,8 @@ pg_create_logical_replication_slot(PG_FUNCTION_ARGS)
temporary,
two_phase,
InvalidXLogRecPtr,
- true);
+ true,
+ onlylocal_data);
values[0] = NameGetDatum(&MyReplicationSlot->data.name);
values[1] = LSNGetDatum(MyReplicationSlot->data.confirmed_flush);
@@ -231,7 +236,7 @@ pg_drop_replication_slot(PG_FUNCTION_ARGS)
Datum
pg_get_replication_slots(PG_FUNCTION_ARGS)
{
-#define PG_GET_REPLICATION_SLOTS_COLS 14
+#define PG_GET_REPLICATION_SLOTS_COLS 15
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -429,6 +434,8 @@ pg_get_replication_slots(PG_FUNCTION_ARGS)
values[i++] = BoolGetDatum(slot_contents.data.two_phase);
+ values[i++] = BoolGetDatum(slot_contents.data.onlylocal_data);
+
Assert(i == PG_GET_REPLICATION_SLOTS_COLS);
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -794,6 +801,7 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot)
temporary,
false,
src_restart_lsn,
+ false,
false);
}
else
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index ceaff097b9..cfdefb1f22 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -374,7 +374,7 @@ WalReceiverMain(void)
"pg_walreceiver_%lld",
(long long int) walrcv_get_backend_pid(wrconn));
- walrcv_create_slot(wrconn, slotname, true, false, 0, NULL);
+ walrcv_create_slot(wrconn, slotname, true, false, 0, NULL, false);
SpinLockAcquire(&walrcv->mutex);
strlcpy(walrcv->slotname, slotname, NAMEDATALEN);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 5a718b1fe9..b826326b98 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -963,12 +963,14 @@ static void
parseCreateReplSlotOptions(CreateReplicationSlotCmd *cmd,
bool *reserve_wal,
CRSSnapshotAction *snapshot_action,
- bool *two_phase)
+ bool *two_phase,
+ bool *onlylocal_data)
{
ListCell *lc;
bool snapshot_action_given = false;
bool reserve_wal_given = false;
bool two_phase_given = false;
+ bool onlylocal_data_given = false;
/* Parse options */
foreach(lc, cmd->options)
@@ -1019,6 +1021,15 @@ parseCreateReplSlotOptions(CreateReplicationSlotCmd *cmd,
two_phase_given = true;
*two_phase = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "only_local") == 0)
+ {
+ if (onlylocal_data_given || cmd->kind != REPLICATION_KIND_LOGICAL)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ onlylocal_data_given = true;
+ *onlylocal_data = defGetBoolean(defel);
+ }
else
elog(ERROR, "unrecognized option: %s", defel->defname);
}
@@ -1035,6 +1046,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
char *slot_name;
bool reserve_wal = false;
bool two_phase = false;
+ bool onlylocal_data = false;
CRSSnapshotAction snapshot_action = CRS_EXPORT_SNAPSHOT;
DestReceiver *dest;
TupOutputState *tstate;
@@ -1044,13 +1056,14 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
Assert(!MyReplicationSlot);
- parseCreateReplSlotOptions(cmd, &reserve_wal, &snapshot_action, &two_phase);
+ parseCreateReplSlotOptions(cmd, &reserve_wal, &snapshot_action, &two_phase,
+ &onlylocal_data);
if (cmd->kind == REPLICATION_KIND_PHYSICAL)
{
ReplicationSlotCreate(cmd->slotname, false,
cmd->temporary ? RS_TEMPORARY : RS_PERSISTENT,
- false);
+ false, false);
}
else
{
@@ -1065,7 +1078,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
*/
ReplicationSlotCreate(cmd->slotname, true,
cmd->temporary ? RS_TEMPORARY : RS_EPHEMERAL,
- two_phase);
+ two_phase, onlylocal_data);
}
if (cmd->kind == REPLICATION_KIND_LOGICAL)
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 6957567264..02a1f96d1e 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1834,7 +1834,7 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "slot_name", "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "only_local", "slot_name", "streaming", "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SET PUBLICATION */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "PUBLICATION"))
{
@@ -3104,7 +3104,7 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
"enabled", "slot_name", "streaming",
- "synchronous_commit", "two_phase");
+ "synchronous_commit", "two_phase", "only_local");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index bf88858171..74a5f2ac0b 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10776,9 +10776,9 @@
proname => 'pg_get_replication_slots', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', prorettype => 'record',
proargtypes => '',
- proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase}',
+ proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,bool}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,only_local}',
prosrc => 'pg_get_replication_slots' },
{ oid => '3786', descr => 'set up a logical replication slot',
proname => 'pg_create_logical_replication_slot', provolatile => 'v',
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 18c291289f..6e3119247c 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -65,6 +65,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool substream; /* Stream in-progress transactions. */
+ bool subonlylocaldata; /* skip copying of remote origin data */
+
char subtwophasestate; /* Stream two-phase transactions */
#ifdef CATALOG_VARLEN /* variable-length fields start here */
@@ -102,6 +104,7 @@ typedef struct Subscription
bool binary; /* Indicates if the subscription wants data in
* binary format */
bool stream; /* Allow streaming in-progress transactions. */
+ bool onlylocaldata; /* Skip copying of remote orging data */
char twophasestate; /* Allow streaming two-phase transactions */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 1097cc9799..82014fe252 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -99,6 +99,8 @@ typedef struct LogicalDecodingContext
*/
bool twophase_opt_given;
+ bool onlylocal_data;
+
/*
* State for writing output.
*/
@@ -138,6 +140,8 @@ extern void LogicalConfirmReceivedLocation(XLogRecPtr lsn);
extern bool filter_prepare_cb_wrapper(LogicalDecodingContext *ctx,
TransactionId xid, const char *gid);
extern bool filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, RepOriginId origin_id);
+extern bool filter_remotedata_cb_wrapper(LogicalDecodingContext *ctx,
+ RepOriginId origin_id);
extern void ResetLogicalStreamingState(void);
extern void UpdateDecodingStats(LogicalDecodingContext *ctx);
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index a16bebf76c..52b5de3eb8 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -106,6 +106,12 @@ typedef void (*LogicalDecodeSequenceCB) (struct LogicalDecodingContext *ctx,
typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ctx,
RepOriginId origin_id);
+/*
+ * Filter remote origin changes.
+ */
+typedef bool (*LogicalDecodeFilterRemoteOriginCB) (struct LogicalDecodingContext *ctx,
+ RepOriginId origin_id);
+
/*
* Called to shutdown an output plugin.
*/
@@ -246,6 +252,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeSequenceCB sequence_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
+ LogicalDecodeFilterRemoteOriginCB filter_remotedata_cb;
LogicalDecodeShutdownCB shutdown_cb;
/* streaming of changes at prepare time */
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index eafedd610a..e8fac6b3f8 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -29,6 +29,7 @@ typedef struct PGOutputData
bool streaming;
bool messages;
bool two_phase;
+ bool onlylocal_data;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 24b30210c3..833d380b0f 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -94,6 +94,8 @@ typedef struct ReplicationSlotPersistentData
*/
bool two_phase;
+ bool onlylocal_data;
+
/* plugin name */
NameData plugin;
} ReplicationSlotPersistentData;
@@ -195,7 +197,8 @@ extern void ReplicationSlotsShmemInit(void);
/* management of individual slots */
extern void ReplicationSlotCreate(const char *name, bool db_specific,
- ReplicationSlotPersistency p, bool two_phase);
+ ReplicationSlotPersistency p, bool two_phase,
+ bool onlylocal_data);
extern void ReplicationSlotPersist(void);
extern void ReplicationSlotDrop(const char *name, bool nowait);
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 92f73a55b8..e62dca9b45 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -183,6 +183,7 @@ typedef struct
bool streaming; /* Streaming of large transactions */
bool twophase; /* Streaming of two-phase transactions at
* prepare time */
+ bool onlylocal_data;
} logical;
} proto;
} WalRcvStreamOptions;
@@ -351,7 +352,8 @@ typedef char *(*walrcv_create_slot_fn) (WalReceiverConn *conn,
bool temporary,
bool two_phase,
CRSSnapshotAction snapshot_action,
- XLogRecPtr *lsn);
+ XLogRecPtr *lsn,
+ bool onlylocal_data);
/*
* walrcv_get_backend_pid_fn
@@ -423,8 +425,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
WalReceiverFunctions->walrcv_receive(conn, buffer, wait_fd)
#define walrcv_send(conn, buffer, nbytes) \
WalReceiverFunctions->walrcv_send(conn, buffer, nbytes)
-#define walrcv_create_slot(conn, slotname, temporary, two_phase, snapshot_action, lsn) \
- WalReceiverFunctions->walrcv_create_slot(conn, slotname, temporary, two_phase, snapshot_action, lsn)
+#define walrcv_create_slot(conn, slotname, temporary, two_phase, snapshot_action, lsn, onlylocal_data) \
+ WalReceiverFunctions->walrcv_create_slot(conn, slotname, temporary, two_phase, snapshot_action, lsn, onlylocal_data)
#define walrcv_get_backend_pid(conn) \
WalReceiverFunctions->walrcv_get_backend_pid(conn)
#define walrcv_exec(conn, exec, nRetTypes, retTypes) \
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index ac468568a1..d4a2ec85e7 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1456,8 +1456,9 @@ pg_replication_slots| SELECT l.slot_name,
l.confirmed_flush_lsn,
l.wal_status,
l.safe_wal_size,
- l.two_phase
- FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase)
+ l.two_phase,
+ l.only_local
+ FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, only_local)
LEFT JOIN pg_database d ON ((l.datoid = d.oid)));
pg_roles| SELECT pg_authid.rolname,
pg_authid.rolsuper,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 80aae83562..dbf75d8b17 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -70,7 +70,11 @@ ALTER SUBSCRIPTION regress_testsub3 ENABLE;
ERROR: cannot enable subscription that does not have a slot name
ALTER SUBSCRIPTION regress_testsub3 REFRESH PUBLICATION;
ERROR: ALTER SUBSCRIPTION ... REFRESH is not allowed for disabled subscriptions
+-- ok - with only_local = true
+CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false, only_local = true);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
DROP SUBSCRIPTION regress_testsub3;
+DROP SUBSCRIPTION regress_testsub4;
-- fail - invalid connection string
ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index bd0f4af1e4..3b65c42142 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -54,7 +54,11 @@ CREATE SUBSCRIPTION regress_testsub3 CONNECTION 'dbname=regress_doesnotexist' PU
ALTER SUBSCRIPTION regress_testsub3 ENABLE;
ALTER SUBSCRIPTION regress_testsub3 REFRESH PUBLICATION;
+-- ok - with only_local = true
+CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (slot_name = NONE, connect = false, only_local = true);
+
DROP SUBSCRIPTION regress_testsub3;
+DROP SUBSCRIPTION regress_testsub4;
-- fail - invalid connection string
ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
diff --git a/src/test/subscription/t/029_circular.pl b/src/test/subscription/t/029_circular.pl
new file mode 100644
index 0000000000..553635ae5d
--- /dev/null
+++ b/src/test/subscription/t/029_circular.pl
@@ -0,0 +1,108 @@
+
+# Copyright (c) 2021-2022, PostgreSQL Global Development Group
+
+# Test circular logical replication.
+#
+# Includes tests for circulation replication using only_local option.
+#
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+###################################################
+# Setup a circulation replication of pub/sub nodes.
+# node_A -> node_B -> node_A
+###################################################
+
+# Initialize nodes
+# node_A
+my $node_A = PostgreSQL::Test::Cluster->new('node_A');
+$node_A->init(allows_streaming => 'logical');
+$node_A->append_conf('postgresql.conf', qq(
+max_prepared_transactions = 10
+logical_decoding_work_mem = 64kB
+));
+$node_A->start;
+# node_B
+my $node_B = PostgreSQL::Test::Cluster->new('node_B');
+$node_B->init(allows_streaming => 'logical');
+$node_B->append_conf('postgresql.conf', qq(
+max_prepared_transactions = 10
+logical_decoding_work_mem = 64kB
+));
+$node_B->start;
+
+# Create tables on node_A
+$node_A->safe_psql('postgres',
+ "CREATE TABLE tab_full (a int PRIMARY KEY)");
+
+# Create the same tables on node_B
+$node_B->safe_psql('postgres',
+ "CREATE TABLE tab_full (a int PRIMARY KEY)");
+
+# Setup logical replication
+# node_A (pub) -> node_B (sub)
+my $node_A_connstr = $node_A->connstr . ' dbname=postgres';
+$node_A->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_A FOR TABLE tab_full");
+my $appname_B = 'tap_sub_B';
+$node_B->safe_psql('postgres', "
+ CREATE SUBSCRIPTION tap_sub_B
+ CONNECTION '$node_A_connstr application_name=$appname_B'
+ PUBLICATION tap_pub_A
+ WITH (only_local = on)");
+
+# node_B (pub) -> node_A (sub)
+my $node_B_connstr = $node_B->connstr . ' dbname=postgres';
+$node_B->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_B FOR TABLE tab_full");
+my $appname_A = 'tap_sub_A';
+$node_A->safe_psql('postgres', "
+ CREATE SUBSCRIPTION tap_sub_A
+ CONNECTION '$node_B_connstr application_name=$appname_A'
+ PUBLICATION tap_pub_B
+ WITH (only_local = on)");
+
+# Wait for subscribers to finish initialization
+$node_A->wait_for_catchup($appname_B);
+$node_B->wait_for_catchup($appname_A);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_A->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+$node_B->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+is(1,1, "Circular replication setup is complete");
+
+my $result;
+
+##########################################################################
+# check that circular replication setup does not cause infinite recursive
+# insertion.
+##########################################################################
+
+# insert a record
+$node_A->safe_psql('postgres', "INSERT INTO tab_full VALUES (11);");
+$node_B->safe_psql('postgres', "INSERT INTO tab_full VALUES (12);");
+
+$node_A->wait_for_catchup($appname_B);
+$node_B->wait_for_catchup($appname_A);
+
+# check that transaction was committed on subscriber(s)
+$result = $node_A->safe_psql('postgres', "SELECT * FROM tab_full;");
+is($result, qq(11
+12), 'Inserted successfully without leading to infinite recursion in circular replication setup');
+$result = $node_B->safe_psql('postgres', "SELECT * FROM tab_full;");
+is($result, qq(11
+12), 'Inserted successfully without leading to infinite recursion in circular replication setup');
+
+# shutdown
+$node_B->stop('fast');
+$node_A->stop('fast');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d9b83f744f..9608cb7bbc 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1370,6 +1370,7 @@ LogicalDecodeCommitCB
LogicalDecodeCommitPreparedCB
LogicalDecodeFilterByOriginCB
LogicalDecodeFilterPrepareCB
+LogicalDecodeFilterRemoteOriginCB
LogicalDecodeMessageCB
LogicalDecodePrepareCB
LogicalDecodeRollbackPreparedCB
--
2.32.0
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Handle infinite recursion in logical replication setup
@ 2022-03-07 03:37 Peter Smith <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Peter Smith @ 2022-03-07 03:37 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: [email protected] <[email protected]>; PostgreSQL Hackers <[email protected]>
FYI, the v2 patch did not apply to HEAD
[postgres@CentOS7-x64 oss_postgres_misc]$ git apply
../patches_misc/v2-0001-Skip-replication-of-non-local-data.patch
--verbose
...
error: patch failed: src/backend/replication/slotfuncs.c:231
error: src/backend/replication/slotfuncs.c: patch does not apply
------
Kind Regards,
Peter Smith.
Fujitsu Australia.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Handle infinite recursion in logical replication setup
@ 2022-03-07 05:44 vignesh C <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 0 replies; 5+ messages in thread
From: vignesh C @ 2022-03-07 05:44 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Tue, Mar 1, 2022 at 4:12 PM [email protected]
<[email protected]> wrote:
>
> Hi Vignesh,
>
> > In logical replication, currently Walsender sends the data that is
> > generated locally and the data that are replicated from other
> > instances. This results in infinite recursion in circular logical
> > replication setup.
>
> Thank you for good explanation. I understand that this fix can be used
> for a bidirectional replication.
>
> > Here there are two problems for the user: a) incremental
> > synchronization of table sending both local data and replicated data
> > by walsender b) Table synchronization of table using copy command
> > sending both local data and replicated data
>
> So you wanted to solve these two problem and currently focused on
> the first one, right? We can check one by one.
>
> > For the first problem "Incremental synchronization of table by
> > Walsender" can be solved by:
> > Currently the locally generated data does not have replication origin
> > associated and the data that has originated from another instance will
> > have a replication origin associated. We could use this information to
> > differentiate locally generated data and replicated data and send only
> > the locally generated data. This "only_local" could be provided as an
> > option while subscription is created:
> > ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=5433'
> > PUBLICATION pub1 with (only_local = on);
>
> Sounds good, but I cannot distinguish whether the assumption will keep.
>
> I played with your patch, but it could not be applied to current master.
> I tested from bd74c40 and I confirmed infinite loop was not appeared.
>
> local_only could not be set from ALTER SUBSCRIPTION command.
> Is it expected?
I felt changing only_local option might be useful for the user while
modifying the subscription like setting it with a different set of
publications. Changes for this are included in the v2 patch attached
at [1].
[2] - https://www.postgresql.org/message-id/CALDaNm0WSo5369pr2eN1obTGBeiJU9cQdF6Ju1sC4hMQNy5BfQ%40mail.gma...
Regards,
Vignesh
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2022-03-07 05:44 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-03-09 06:07 [PATCH v50 4/7] Shared-memory based stats collector Kyotaro Horiguchi <[email protected]>
2022-03-01 10:42 RE: Handle infinite recursion in logical replication setup [email protected] <[email protected]>
2022-03-02 16:12 ` Re: Handle infinite recursion in logical replication setup vignesh C <[email protected]>
2022-03-07 03:37 ` Re: Handle infinite recursion in logical replication setup Peter Smith <[email protected]>
2022-03-07 05:44 ` Re: Handle infinite recursion in logical replication setup vignesh C <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox