Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nbE8J-0005gG-05 for pgsql-hackers@arkaria.postgresql.org; Mon, 04 Apr 2022 04:16:04 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1nbE8H-0006VN-6d for pgsql-hackers@arkaria.postgresql.org; Mon, 04 Apr 2022 04:16:01 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nbE8E-0006V6-Uu for pgsql-hackers@lists.postgresql.org; Mon, 04 Apr 2022 04:16:00 +0000 Received: from out4-smtp.messagingengine.com ([66.111.4.28]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nbE7e-0001lr-Lh for pgsql-hackers@lists.postgresql.org; Mon, 04 Apr 2022 04:15:58 +0000 Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 9E8785C00E8; Mon, 4 Apr 2022 00:15:19 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Mon, 04 Apr 2022 00:15:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=anarazel.de; h= cc:cc:content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; bh=X2ZXMvaOxkH0Ul0p4jo3KqNL2UYhsDVozI5Bsn OlEB0=; b=23MBA0wo6HSeGbIJHMZu7JAdIKJQQHMPRbiNhvguWVOcVq1TYrA/8A 9Fc/VZe1DoE07r1AM5MHq2rCqMhShpMZvFHPZAZxISyPjZabV+9ZzA5vBSM0ryDh fFC0oX3X0BjeG/uptDdNt4ZsmGS3oD1WNfxMdj0+VDYWUDbPVOpw9OkUnoG8SU28 DcWdFf0dlnR/NtHmYcnlV++hh+QnkY1zox4WBpWiEFAcY6uUkWLe35E+LSMj/1w+ yMSkP7I5ZvsaWNndeDMPMkIX3eS0j8D2fWyqaE4lyFMiWMnh9YZXE1Zz4ggCJBtl KqPBsdsY12Gk5W6Hh3zgrnWcw6cgHcfQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=X2ZXMvaOxkH0Ul0p4 jo3KqNL2UYhsDVozI5BsnOlEB0=; b=e06j7XmKzJ+IZ3fpIy9mnI1lHvS/CHQPS 4GMOL2Kwv39hNWTeyU5IH6N8YRFlTCViRpEfm9qTeEQaviIjvVUa+AuK6rJayAPO 4Zw4O6nHX34VA4ktB7qsWZDu+6jVKaiYN1sChCabC9i3dfr8bPKkUEysMCgCR3/7 J4LY16EZ30wAAmzDAfPQJNep5hLQWyoIv+4djcxCPatT7pA8DTKWUDiM9KaDhZAX CJHX3ynxFoWWPlpEk0HhzdM8HhLYkOuhrhdY8VJ0pIHwIqNkgBvjauKjy8cuc9jm Ps+9F151uSWzs3FYlyhSMSS66WB0bguj4slymDLGLKj3UxY6fjvtw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudejuddgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesmhdtre ertddtvdenucfhrhhomheptehnughrvghsucfhrhgvuhhnugcuoegrnhgurhgvshesrghn rghrrgiivghlrdguvgeqnecuggftrfgrthhtvghrnhephfduhedtieekjeethfegtedtle egffegteelleeugeevheekleehgfehieffveejnecuffhomhgrihhnpehpohhsthhgrhgv shhqlhdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfh hrohhmpegrnhgurhgvshesrghnrghrrgiivghlrdguvg X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 4 Apr 2022 00:15:17 -0400 (EDT) Date: Sun, 3 Apr 2022 21:15:16 -0700 From: Andres Freund To: Kyotaro Horiguchi Cc: melanieplageman@gmail.com, ibrar.ahmad@gmail.com, masao.fujii@oss.nttdata.com, gkokolatos@protonmail.com, pgsql-hackers@lists.postgresql.org Subject: Re: shared-memory based stats collector - v68 Message-ID: <20220404041516.cctrvpadhuriawlq@alap3.anarazel.de> References: <20220317073652.kz3uuh6le6vbfod4@alap3.anarazel.de> <20220325.142256.1649716600998659084.horikyota.ntt@gmail.com> <20220325.172418.1055357550509939415.horikyota.ntt@gmail.com> <20220402082102.ojdd2e6cobsmljvh@alap3.anarazel.de> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="c4n45bccvxafocvc" Content-Disposition: inline In-Reply-To: <20220402082102.ojdd2e6cobsmljvh@alap3.anarazel.de> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --c4n45bccvxafocvc Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, New version of the shared memory stats patchset. Most important changes: - It's now "cumulative statistics system", as discussed at [1]. This basically is now the term that all the references to the "stats collector" are replaced with. Looks much better than "activity statistics" imo. The STATS_COLLECTOR is now named STATS_CUMULATIVE. I tried to find all references to either collector or "activity statistics", but in all likelihood I didn't get them all. - updated docs (significantly edited version of the version Kyotaro posted a few days ago) - significantly improved test coverage - pgstat*.c are nearly completely covered. While pgstatsfuncs.c coverage has increased, it is not great - but there's already so much more coverage, that I think it's good enough for now. Thanks to Melanie for help with this! - largely cleaned up inconsisten function / type naming. Everything now / again is under the PgStats_ prefix, except for statistics in shared memory, which is prefixed with PgStatsShared_. I think we should go further and add at least a PgStatsPending_ namespace, but that requires touching plenty code that didn't need to be touched so far, so it'll have to be task for another release. - As discussed in [2] I added a patch at the start of the queue to clean up the inconsistent function header comments conventions. - pgstat.c is further split. Two new files: pgstat_xact.c and pgstat_shmem.c (wrote an email about this a few days ago, without sending the patches) - Split out as much as I could into separate commits. - Cleaned up autovacuum.c changes - mostly removing more obsolted code - code, comment polishing Still todo: - docs need review - finish writing architectural comment atop pgstat.c - fix the bug around pgstat_report_stat() I wrote about at [3] - collect who reviewed earlier revisions - choose what conditions for stats file reset we want - I'm wondering if the solution for replication slot names on disk is too narrow, and instead we should have a more general "serialize" / "deserialize" callback. But we can easily do that later as well... There's a bit more inconsistency around function naming. Right now all callbacks are pgstat_$kind_$action_cb, but most of the rest of pgstat is pgstat_$action_$kind. But somehow it "feels" wrong for the callbacks - there's also a bunch of functions already named similarly, but that's partially my fault in commits in the past. There are a lot of copies of "Permission checking for this function is managed through the normal GRANT system." in the pre-existing code. Aren't they completely bogus? None of the functions commented upon like that is actually exposed to SQL! Please take a look! Greetings, Andres Freund [1] https://www.postgresql.org/message-id/20220308205351.2xcn6k4x5yivcxyd%40alap3.anarazel.de [2] https://www.postgresql.org/message-id/20220329191727.mzzwbl7udhpq7pmf%40alap3.anarazel.de [3] https://www.postgresql.org/message-id/20220402081648.kbapqdxi2rr3ha3w@alap3.anarazel.de --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0001-pgstat-consistent-function-header-formatting.patch" From b19019774911f4ba49047ba837dfc3d1e3930b19 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Tue, 29 Mar 2022 21:40:32 -0700 Subject: [PATCH v68 01/31] pgstat: consistent function header formatting. Author: Andres Freund Discussion: https://postgr.es/m/20220329191727.mzzwbl7udhpq7pmf@alap3.anarazel.de --- src/backend/postmaster/pgstat.c | 636 ++++++------------ src/backend/utils/activity/pgstat_archiver.c | 9 +- src/backend/utils/activity/pgstat_bgwriter.c | 7 +- .../utils/activity/pgstat_checkpointer.c | 7 +- src/backend/utils/activity/pgstat_database.c | 75 +-- src/backend/utils/activity/pgstat_function.c | 3 +- src/backend/utils/activity/pgstat_relation.c | 77 +-- src/backend/utils/activity/pgstat_replslot.c | 34 +- src/backend/utils/activity/pgstat_slru.c | 26 +- .../utils/activity/pgstat_subscription.c | 27 +- src/backend/utils/activity/pgstat_wal.c | 7 +- 11 files changed, 300 insertions(+), 608 deletions(-) diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index c10311e0366..ef1cba61a6f 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/postmaster/pgstat.c @@ -240,14 +240,11 @@ static bool pgstat_is_shutdown = false; * ------------------------------------------------------------ */ -/* ---------- - * pgstat_init() - - * - * Called from postmaster at startup. Create the resources required - * by the statistics collector process. If unable to do so, do not - * fail --- better to let the postmaster start with stats collection - * disabled. - * ---------- +/* + * Called from postmaster at startup. Create the resources required + * by the statistics collector process. If unable to do so, do not + * fail --- better to let the postmaster start with stats collection + * disabled. */ void pgstat_init(void) @@ -579,8 +576,6 @@ pgstat_reset_remove_files(const char *directory) } /* - * pgstat_reset_all() - - * * Remove the stats files. This is currently used only if WAL * recovery is needed after a crash. */ @@ -594,8 +589,6 @@ pgstat_reset_all(void) #ifdef EXEC_BACKEND /* - * pgstat_forkexec() - - * * Format up the arglist for, then fork and exec, statistics collector process */ static pid_t @@ -617,14 +610,12 @@ pgstat_forkexec(void) /* - * pgstat_start() - + * Called from postmaster at startup or after an existing collector + * died. Attempt to fire up a fresh statistics collector. * - * Called from postmaster at startup or after an existing collector - * died. Attempt to fire up a fresh statistics collector. + * Returns PID of child process, or 0 if fail. * - * Returns PID of child process, or 0 if fail. - * - * Note: if fail, we will be called again from the postmaster main loop. + * Note: if fail, we will be called again from the postmaster main loop. */ int pgstat_start(void) @@ -727,14 +718,11 @@ pgstat_shutdown_hook(int code, Datum arg) #endif } -/* ---------- - * pgstat_initialize() - +/* + * Initialize pgstats state, and set up our on-proc-exit hook. Called from + * BaseInit(). * - * Initialize pgstats state, and set up our on-proc-exit hook. Called from - * BaseInit(). - * - * NOTE: MyDatabaseId isn't set yet; so the shutdown hook has to be careful. - * ---------- + * NOTE: MyDatabaseId isn't set yet; so the shutdown hook has to be careful. */ void pgstat_initialize(void) @@ -757,11 +745,8 @@ pgstat_initialize(void) * ------------------------------------------------------------ */ -/* ---------- - * AtEOXact_PgStat - * - * Called from access/transam/xact.c at top-level transaction commit/abort. - * ---------- +/* + * Called from access/transam/xact.c at top-level transaction commit/abort. */ void AtEOXact_PgStat(bool isCommit, bool parallel) @@ -785,11 +770,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel) pgstat_clear_snapshot(); } -/* ---------- - * AtEOSubXact_PgStat - * - * Called from access/transam/xact.c at subtransaction commit/abort. - * ---------- +/* + * Called from access/transam/xact.c at subtransaction commit/abort. */ void AtEOSubXact_PgStat(bool isCommit, int nestDepth) @@ -811,8 +793,7 @@ AtEOSubXact_PgStat(bool isCommit, int nestDepth) } /* - * AtPrepare_PgStat - * Save the transactional stats state at 2PC transaction prepare. + * Save the transactional stats state at 2PC transaction prepare. */ void AtPrepare_PgStat(void) @@ -830,8 +811,7 @@ AtPrepare_PgStat(void) } /* - * PostPrepare_PgStat - * Clean up after successful PREPARE. + * Clean up after successful PREPARE. * * Note: AtEOXact_PgStat is not called during PREPARE. */ @@ -858,15 +838,12 @@ PostPrepare_PgStat(void) pgstat_clear_snapshot(); } -/* ---------- - * pgstat_clear_snapshot() - +/* + * Discard any data collected in the current transaction. Any subsequent + * request will cause new snapshots to be read. * - * Discard any data collected in the current transaction. Any subsequent - * request will cause new snapshots to be read. - * - * This is also invoked during transaction commit or abort to discard - * the no-longer-wanted snapshot. - * ---------- + * This is also invoked during transaction commit or abort to discard + * the no-longer-wanted snapshot. */ void pgstat_clear_snapshot(void) @@ -920,19 +897,16 @@ pgstat_xact_stack_level_get(int nest_level) * ------------------------------------------------------------ */ -/* ---------- - * pgstat_report_stat() - +/* + * Must be called by processes that performs DML: tcop/postgres.c, logical + * receiver processes, SPI worker, etc. to send the so far collected + * per-table and function usage statistics to the collector. Note that this + * is called only when not within a transaction, so it is fair to use + * transaction stop time as an approximation of current time. * - * Must be called by processes that performs DML: tcop/postgres.c, logical - * receiver processes, SPI worker, etc. to send the so far collected - * per-table and function usage statistics to the collector. Note that this - * is called only when not within a transaction, so it is fair to use - * transaction stop time as an approximation of current time. - * - * "disconnect" is "true" only for the last call before the backend - * exits. This makes sure that no data is lost and that interrupted - * sessions are reported correctly. - * ---------- + * "disconnect" is "true" only for the last call before the backend + * exits. This makes sure that no data is lost and that interrupted + * sessions are reported correctly. */ void pgstat_report_stat(bool disconnect) @@ -979,11 +953,8 @@ pgstat_report_stat(bool disconnect) pgstat_send_slru(); } -/* ---------- - * pgstat_vacuum_stat() - - * - * Will tell the collector about objects he can get rid of. - * ---------- +/* + * Will tell the collector about objects he can get rid of. */ void pgstat_vacuum_stat(void) @@ -1201,14 +1172,11 @@ pgstat_vacuum_stat(void) } } -/* ---------- - * pgstat_collect_oids() - - * - * Collect the OIDs of all objects listed in the specified system catalog - * into a temporary hash table. Caller should hash_destroy the result - * when done with it. (However, we make the table in CurrentMemoryContext - * so that it will be freed properly in event of an error.) - * ---------- +/* + * Collect the OIDs of all objects listed in the specified system catalog + * into a temporary hash table. Caller should hash_destroy the result + * when done with it. (However, we make the table in CurrentMemoryContext + * so that it will be freed properly in event of an error.) */ static HTAB * pgstat_collect_oids(Oid catalogid, AttrNumber anum_oid) @@ -1250,14 +1218,11 @@ pgstat_collect_oids(Oid catalogid, AttrNumber anum_oid) return htab; } -/* ---------- - * pgstat_reset_counters() - +/* + * Tell the statistics collector to reset counters for our database. * - * Tell the statistics collector to reset counters for our database. - * - * Permission checking for this function is managed through the normal - * GRANT system. - * ---------- + * Permission checking for this function is managed through the normal + * GRANT system. */ void pgstat_reset_counters(void) @@ -1272,14 +1237,11 @@ pgstat_reset_counters(void) pgstat_send(&msg, sizeof(msg)); } -/* ---------- - * pgstat_reset_single_counter() - +/* + * Tell the statistics collector to reset a single counter. * - * Tell the statistics collector to reset a single counter. - * - * Permission checking for this function is managed through the normal - * GRANT system. - * ---------- + * Permission checking for this function is managed through the normal + * GRANT system. */ void pgstat_reset_single_counter(Oid objoid, PgStat_Single_Reset_Type type) @@ -1297,14 +1259,11 @@ pgstat_reset_single_counter(Oid objoid, PgStat_Single_Reset_Type type) pgstat_send(&msg, sizeof(msg)); } -/* ---------- - * pgstat_reset_shared_counters() - +/* + * Tell the statistics collector to reset cluster-wide shared counters. * - * Tell the statistics collector to reset cluster-wide shared counters. - * - * Permission checking for this function is managed through the normal - * GRANT system. - * ---------- + * Permission checking for this function is managed through the normal + * GRANT system. */ void pgstat_reset_shared_counters(const char *target) @@ -1330,11 +1289,8 @@ pgstat_reset_shared_counters(const char *target) pgstat_send(&msg, sizeof(msg)); } -/* ---------- - * pgstat_ping() - - * - * Send some junk data to the collector to increase traffic. - * ---------- +/* + * Send some junk data to the collector to increase traffic. */ void pgstat_ping(void) @@ -1348,11 +1304,8 @@ pgstat_ping(void) pgstat_send(&msg, sizeof(msg)); } -/* ---------- - * pgstat_send_inquiry() - - * - * Notify collector that we need fresh data. - * ---------- +/* + * Notify collector that we need fresh data. */ static void pgstat_send_inquiry(TimestampTz clock_time, TimestampTz cutoff_time, Oid databaseid) @@ -1366,14 +1319,11 @@ pgstat_send_inquiry(TimestampTz clock_time, TimestampTz cutoff_time, Oid databas pgstat_send(&msg, sizeof(msg)); } -/* ---------- - * pgstat_fetch_stat_dbentry() - - * - * Support function for the SQL-callable pgstat* functions. Returns - * the collected statistics for one database or NULL. NULL doesn't mean - * that the database doesn't exist, it is just not yet known by the - * collector, so the caller is better off to report ZERO instead. - * ---------- +/* + * Support function for the SQL-callable pgstat* functions. Returns + * the collected statistics for one database or NULL. NULL doesn't mean + * that the database doesn't exist, it is just not yet known by the + * collector, so the caller is better off to report ZERO instead. */ PgStat_StatDBEntry * pgstat_fetch_stat_dbentry(Oid dbid) @@ -1393,12 +1343,8 @@ pgstat_fetch_stat_dbentry(Oid dbid) } /* - * --------- - * pgstat_fetch_global() - - * - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the global statistics struct. - * --------- + * Support function for the SQL-callable pgstat* functions. Returns + * a pointer to the global statistics struct. */ PgStat_GlobalStats * pgstat_fetch_global(void) @@ -1408,14 +1354,11 @@ pgstat_fetch_global(void) return &globalStats; } -/* ---------- - * pgstat_fetch_stat_tabentry() - - * - * Support function for the SQL-callable pgstat* functions. Returns - * the collected statistics for one table or NULL. NULL doesn't mean - * that the table doesn't exist, it is just not yet known by the - * collector, so the caller is better off to report ZERO instead. - * ---------- +/* + * Support function for the SQL-callable pgstat* functions. Returns + * the collected statistics for one table or NULL. NULL doesn't mean + * that the table doesn't exist, it is just not yet known by the + * collector, so the caller is better off to report ZERO instead. */ PgStat_StatTabEntry * pgstat_fetch_stat_tabentry(Oid relid) @@ -1466,12 +1409,9 @@ pgstat_fetch_stat_tabentry(Oid relid) } -/* ---------- - * pgstat_fetch_stat_funcentry() - - * - * Support function for the SQL-callable pgstat* functions. Returns - * the collected statistics for one function or NULL. - * ---------- +/* + * Support function for the SQL-callable pgstat* functions. Returns + * the collected statistics for one function or NULL. */ PgStat_StatFuncEntry * pgstat_fetch_stat_funcentry(Oid func_id) @@ -1495,12 +1435,8 @@ pgstat_fetch_stat_funcentry(Oid func_id) } /* - * --------- - * pgstat_fetch_stat_archiver() - - * - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the archiver statistics struct. - * --------- + * Support function for the SQL-callable pgstat* functions. Returns + * a pointer to the archiver statistics struct. */ PgStat_ArchiverStats * pgstat_fetch_stat_archiver(void) @@ -1511,12 +1447,8 @@ pgstat_fetch_stat_archiver(void) } /* - * --------- - * pgstat_fetch_stat_bgwriter() - - * - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the bgwriter statistics struct. - * --------- + * Support function for the SQL-callable pgstat* functions. Returns + * a pointer to the bgwriter statistics struct. */ PgStat_BgWriterStats * pgstat_fetch_stat_bgwriter(void) @@ -1527,12 +1459,8 @@ pgstat_fetch_stat_bgwriter(void) } /* - * --------- - * pgstat_fetch_stat_checkpointer() - - * - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the checkpointer statistics struct. - * --------- + * Support function for the SQL-callable pgstat* functions. Returns + * a pointer to the checkpointer statistics struct. */ PgStat_CheckpointerStats * pgstat_fetch_stat_checkpointer(void) @@ -1543,12 +1471,8 @@ pgstat_fetch_stat_checkpointer(void) } /* - * --------- - * pgstat_fetch_stat_wal() - - * - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the WAL statistics struct. - * --------- + * Support function for the SQL-callable pgstat* functions. Returns + * a pointer to the WAL statistics struct. */ PgStat_WalStats * pgstat_fetch_stat_wal(void) @@ -1559,12 +1483,8 @@ pgstat_fetch_stat_wal(void) } /* - * --------- - * pgstat_fetch_slru() - - * - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the slru statistics struct. - * --------- + * Support function for the SQL-callable pgstat* functions. Returns + * a pointer to the slru statistics struct. */ PgStat_SLRUStats * pgstat_fetch_slru(void) @@ -1575,12 +1495,8 @@ pgstat_fetch_slru(void) } /* - * --------- - * pgstat_fetch_replslot() - - * - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the replication slot statistics struct. - * --------- + * Support function for the SQL-callable pgstat* functions. Returns + * a pointer to the replication slot statistics struct. */ PgStat_StatReplSlotEntry * pgstat_fetch_replslot(NameData slotname) @@ -1591,12 +1507,8 @@ pgstat_fetch_replslot(NameData slotname) } /* - * --------- - * pgstat_fetch_stat_subscription() - - * - * Support function for the SQL-callable pgstat* functions. Returns - * the collected statistics for one subscription or NULL. - * --------- + * Support function for the SQL-callable pgstat* functions. Returns + * the collected statistics for one subscription or NULL. */ PgStat_StatSubEntry * pgstat_fetch_stat_subscription(Oid subid) @@ -1613,11 +1525,8 @@ pgstat_fetch_stat_subscription(Oid subid) * ------------------------------------------------------------ */ -/* ---------- - * pgstat_setup_memcxt() - - * - * Create pgStatLocalContext, if not already done. - * ---------- +/* + * Create pgStatLocalContext, if not already done. */ static void pgstat_setup_memcxt(void) @@ -1641,11 +1550,8 @@ pgstat_assert_is_up(void) } #endif -/* ---------- - * pgstat_setheader() - - * - * Set common header fields in a statistics message - * ---------- +/* + * Set common header fields in a statistics message */ void pgstat_setheader(PgStat_MsgHdr *hdr, StatMsgType mtype) @@ -1654,11 +1560,8 @@ pgstat_setheader(PgStat_MsgHdr *hdr, StatMsgType mtype) } -/* ---------- - * pgstat_send() - - * - * Send out one statistics message to the collector - * ---------- +/* + * Send out one statistics message to the collector */ void pgstat_send(void *msg, int len) @@ -1685,14 +1588,11 @@ pgstat_send(void *msg, int len) #endif } -/* ---------- - * PgstatCollectorMain() - +/* + * Start up the statistics collector process. This is the body of the + * postmaster child process. * - * Start up the statistics collector process. This is the body of the - * postmaster child process. - * - * The argc/argv parameters are valid only in EXEC_BACKEND case. - * ---------- + * The argc/argv parameters are valid only in EXEC_BACKEND case. */ NON_EXEC_STATIC void PgstatCollectorMain(int argc, char *argv[]) @@ -2131,14 +2031,11 @@ pgstat_get_tab_entry(PgStat_StatDBEntry *dbentry, Oid tableoid, bool create) return result; } -/* ---------- - * pgstat_get_replslot_entry - * +/* * Return the entry of replication slot stats with the given name. Return * NULL if not found and the caller didn't request to create it. * * create tells whether to create the new slot entry if it is not found. - * ---------- */ static PgStat_StatReplSlotEntry * pgstat_get_replslot_entry(NameData name, bool create) @@ -2187,11 +2084,8 @@ pgstat_get_replslot_entry(NameData name, bool create) return slotent; } -/* ---------- - * pgstat_reset_replslot - * +/* * Reset the given replication slot stats. - * ---------- */ static void pgstat_reset_replslot(PgStat_StatReplSlotEntry *slotent, TimestampTz ts) @@ -2208,13 +2102,10 @@ pgstat_reset_replslot(PgStat_StatReplSlotEntry *slotent, TimestampTz ts) slotent->stat_reset_timestamp = ts; } -/* ---------- - * pgstat_get_subscription_entry - * +/* * Return the subscription statistics entry with the given subscription OID. * If no subscription entry exists, initialize it, if the create parameter is * true. Else, return NULL. - * ---------- */ static PgStat_StatSubEntry * pgstat_get_subscription_entry(Oid subid, bool create) @@ -2256,11 +2147,8 @@ pgstat_get_subscription_entry(Oid subid, bool create) return subentry; } -/* ---------- - * pgstat_reset_subscription - * +/* * Reset the given subscription stats. - * ---------- */ static void pgstat_reset_subscription(PgStat_StatSubEntry *subentry, TimestampTz ts) @@ -2276,19 +2164,17 @@ pgstat_reset_subscription(PgStat_StatSubEntry *subentry, TimestampTz ts) * ------------------------------------------------------------ */ -/* ---------- - * pgstat_write_statsfiles() - - * Write the global statistics file, as well as requested DB files. +/* + * Write the global statistics file, as well as requested DB files. * - * 'permanent' specifies writing to the permanent files not temporary ones. - * When true (happens only when the collector is shutting down), also remove - * the temporary files so that backends starting up under a new postmaster - * can't read old data before the new collector is ready. + * 'permanent' specifies writing to the permanent files not temporary ones. + * When true (happens only when the collector is shutting down), also remove + * the temporary files so that backends starting up under a new postmaster + * can't read old data before the new collector is ready. * - * When 'allDbs' is false, only the requested databases (listed in - * pending_write_requests) will be written; otherwise, all databases - * will be written. - * ---------- + * When 'allDbs' is false, only the requested databases (listed in + * pending_write_requests) will be written; otherwise, all databases + * will be written. */ static void pgstat_write_statsfiles(bool permanent, bool allDbs) @@ -2475,15 +2361,13 @@ get_dbstat_filename(bool permanent, bool tempname, Oid databaseid, elog(ERROR, "overlength pgstat path"); } -/* ---------- - * pgstat_write_db_statsfile() - - * Write the stat file for a single database. +/* + * Write the stat file for a single database. * - * If writing to the permanent file (happens when the collector is - * shutting down only), remove the temporary file so that backends - * starting up under a new postmaster can't read the old data before - * the new collector is ready. - * ---------- + * If writing to the permanent file (happens when the collector is + * shutting down only), remove the temporary file so that backends + * starting up under a new postmaster can't read the old data before + * the new collector is ready. */ static void pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent) @@ -2588,25 +2472,22 @@ pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent) } } -/* ---------- - * pgstat_read_statsfiles() - +/* + * Reads in some existing statistics collector files and returns the + * databases hash table that is the top level of the data. * - * Reads in some existing statistics collector files and returns the - * databases hash table that is the top level of the data. + * If 'onlydb' is not InvalidOid, it means we only want data for that DB + * plus the shared catalogs ("DB 0"). We'll still populate the DB hash + * table for all databases, but we don't bother even creating table/function + * hash tables for other databases. * - * If 'onlydb' is not InvalidOid, it means we only want data for that DB - * plus the shared catalogs ("DB 0"). We'll still populate the DB hash - * table for all databases, but we don't bother even creating table/function - * hash tables for other databases. + * 'permanent' specifies reading from the permanent files not temporary ones. + * When true (happens only when the collector is starting up), remove the + * files after reading; the in-memory status is now authoritative, and the + * files would be out of date in case somebody else reads them. * - * 'permanent' specifies reading from the permanent files not temporary ones. - * When true (happens only when the collector is starting up), remove the - * files after reading; the in-memory status is now authoritative, and the - * files would be out of date in case somebody else reads them. - * - * If a 'deep' read is requested, table/function stats are read, otherwise - * the table/function hash tables remain empty. - * ---------- + * If a 'deep' read is requested, table/function stats are read, otherwise + * the table/function hash tables remain empty. */ static HTAB * pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep) @@ -2936,19 +2817,16 @@ done: } -/* ---------- - * pgstat_read_db_statsfile() - +/* + * Reads in the existing statistics collector file for the given database, + * filling the passed-in tables and functions hash tables. * - * Reads in the existing statistics collector file for the given database, - * filling the passed-in tables and functions hash tables. + * As in pgstat_read_statsfiles, if the permanent file is requested, it is + * removed after reading. * - * As in pgstat_read_statsfiles, if the permanent file is requested, it is - * removed after reading. - * - * Note: this code has the ability to skip storing per-table or per-function - * data, if NULL is passed for the corresponding hashtable. That's not used - * at the moment though. - * ---------- + * Note: this code has the ability to skip storing per-table or per-function + * data, if NULL is passed for the corresponding hashtable. That's not used + * at the moment though. */ static void pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash, @@ -3095,23 +2973,20 @@ done: } } -/* ---------- - * pgstat_read_db_statsfile_timestamp() - +/* + * Attempt to determine the timestamp of the last db statfile write. + * Returns true if successful; the timestamp is stored in *ts. The caller must + * rely on timestamp stored in *ts iff the function returns true. * - * Attempt to determine the timestamp of the last db statfile write. - * Returns true if successful; the timestamp is stored in *ts. The caller must - * rely on timestamp stored in *ts iff the function returns true. + * This needs to be careful about handling databases for which no stats file + * exists, such as databases without a stat entry or those not yet written: * - * This needs to be careful about handling databases for which no stats file - * exists, such as databases without a stat entry or those not yet written: + * - if there's a database entry in the global file, return the corresponding + * stats_timestamp value. * - * - if there's a database entry in the global file, return the corresponding - * stats_timestamp value. - * - * - if there's no db stat entry (e.g. for a new or inactive database), - * there's no stats_timestamp value, but also nothing to write so we return - * the timestamp of the global statfile. - * ---------- + * - if there's no db stat entry (e.g. for a new or inactive database), + * there's no stats_timestamp value, but also nothing to write so we return + * the timestamp of the global statfile. */ static bool pgstat_read_db_statsfile_timestamp(Oid databaseid, bool permanent, @@ -3424,11 +3299,8 @@ backend_read_statsfile(void) pgStatDBHash = pgstat_read_statsfiles(MyDatabaseId, false, true); } -/* ---------- - * pgstat_write_statsfile_needed() - - * - * Do we need to write out any stats files? - * ---------- +/* + * Do we need to write out any stats files? */ static bool pgstat_write_statsfile_needed(void) @@ -3440,11 +3312,8 @@ pgstat_write_statsfile_needed(void) return false; } -/* ---------- - * pgstat_db_requested() - - * - * Checks whether stats for a particular DB need to be written to a file. - * ---------- +/* + * Checks whether stats for a particular DB need to be written to a file. */ static bool pgstat_db_requested(Oid databaseid) @@ -3471,11 +3340,8 @@ pgstat_db_requested(Oid databaseid) * ------------------------------------------------------------ */ -/* ---------- - * pgstat_recv_inquiry() - - * - * Process stat inquiry requests. - * ---------- +/* + * Process stat inquiry requests. */ static void pgstat_recv_inquiry(PgStat_MsgInquiry *msg, int len) @@ -3563,11 +3429,8 @@ pgstat_recv_inquiry(PgStat_MsgInquiry *msg, int len) msg->databaseid); } -/* ---------- - * pgstat_recv_tabstat() - - * - * Count what the backend has done. - * ---------- +/* + * Count what the backend has done. */ static void pgstat_recv_tabstat(PgStat_MsgTabstat *msg, int len) @@ -3680,11 +3543,8 @@ pgstat_recv_tabstat(PgStat_MsgTabstat *msg, int len) } } -/* ---------- - * pgstat_recv_tabpurge() - - * - * Arrange for dead table removal. - * ---------- +/* + * Arrange for dead table removal. */ static void pgstat_recv_tabpurge(PgStat_MsgTabpurge *msg, int len) @@ -3712,11 +3572,8 @@ pgstat_recv_tabpurge(PgStat_MsgTabpurge *msg, int len) } } -/* ---------- - * pgstat_recv_dropdb() - - * - * Arrange for dead database removal - * ---------- +/* + * Arrange for dead database removal */ static void pgstat_recv_dropdb(PgStat_MsgDropdb *msg, int len) @@ -3754,11 +3611,8 @@ pgstat_recv_dropdb(PgStat_MsgDropdb *msg, int len) } } -/* ---------- - * pgstat_recv_resetcounter() - - * - * Reset the statistics for the specified database. - * ---------- +/* + * Reset the statistics for the specified database. */ static void pgstat_recv_resetcounter(PgStat_MsgResetcounter *msg, int len) @@ -3792,11 +3646,8 @@ pgstat_recv_resetcounter(PgStat_MsgResetcounter *msg, int len) reset_dbentry_counters(dbentry); } -/* ---------- - * pgstat_recv_resetsharedcounter() - - * - * Reset some shared statistics of the cluster. - * ---------- +/* + * Reset some shared statistics of the cluster. */ static void pgstat_recv_resetsharedcounter(PgStat_MsgResetsharedcounter *msg, int len) @@ -3829,12 +3680,9 @@ pgstat_recv_resetsharedcounter(PgStat_MsgResetsharedcounter *msg, int len) */ } -/* ---------- - * pgstat_recv_resetsinglecounter() - - * - * Reset a statistics for a single object, which may be of current - * database or shared across all databases in the cluster. - * ---------- +/* + * Reset a statistics for a single object, which may be of current + * database or shared across all databases in the cluster. */ static void pgstat_recv_resetsinglecounter(PgStat_MsgResetsinglecounter *msg, int len) @@ -3861,11 +3709,8 @@ pgstat_recv_resetsinglecounter(PgStat_MsgResetsinglecounter *msg, int len) HASH_REMOVE, NULL); } -/* ---------- - * pgstat_recv_resetslrucounter() - - * - * Reset some SLRU statistics of the cluster. - * ---------- +/* + * Reset some SLRU statistics of the cluster. */ static void pgstat_recv_resetslrucounter(PgStat_MsgResetslrucounter *msg, int len) @@ -3884,11 +3729,8 @@ pgstat_recv_resetslrucounter(PgStat_MsgResetslrucounter *msg, int len) } } -/* ---------- - * pgstat_recv_resetreplslotcounter() - - * - * Reset some replication slot statistics of the cluster. - * ---------- +/* + * Reset some replication slot statistics of the cluster. */ static void pgstat_recv_resetreplslotcounter(PgStat_MsgResetreplslotcounter *msg, @@ -3929,11 +3771,8 @@ pgstat_recv_resetreplslotcounter(PgStat_MsgResetreplslotcounter *msg, } } -/* ---------- - * pgstat_recv_resetsubcounter() - - * - * Reset some subscription statistics of the cluster. - * ---------- +/* + * Reset some subscription statistics of the cluster. */ static void pgstat_recv_resetsubcounter(PgStat_MsgResetsubcounter *msg, int len) @@ -3974,11 +3813,8 @@ pgstat_recv_resetsubcounter(PgStat_MsgResetsubcounter *msg, int len) } } -/* ---------- - * pgstat_recv_autovac() - - * - * Process an autovacuum signaling message. - * ---------- +/* + * Process an autovacuum signaling message. */ static void pgstat_recv_autovac(PgStat_MsgAutovacStart *msg, int len) @@ -3993,11 +3829,8 @@ pgstat_recv_autovac(PgStat_MsgAutovacStart *msg, int len) dbentry->last_autovac_time = msg->m_start_time; } -/* ---------- - * pgstat_recv_vacuum() - - * - * Process a VACUUM message. - * ---------- +/* + * Process a VACUUM message. */ static void pgstat_recv_vacuum(PgStat_MsgVacuum *msg, int len) @@ -4039,11 +3872,8 @@ pgstat_recv_vacuum(PgStat_MsgVacuum *msg, int len) } } -/* ---------- - * pgstat_recv_analyze() - - * - * Process an ANALYZE message. - * ---------- +/* + * Process an ANALYZE message. */ static void pgstat_recv_analyze(PgStat_MsgAnalyze *msg, int len) @@ -4081,11 +3911,8 @@ pgstat_recv_analyze(PgStat_MsgAnalyze *msg, int len) } } -/* ---------- - * pgstat_recv_archiver() - - * - * Process a ARCHIVER message. - * ---------- +/* + * Process a ARCHIVER message. */ static void pgstat_recv_archiver(PgStat_MsgArchiver *msg, int len) @@ -4108,11 +3935,8 @@ pgstat_recv_archiver(PgStat_MsgArchiver *msg, int len) } } -/* ---------- - * pgstat_recv_bgwriter() - - * - * Process a BGWRITER message. - * ---------- +/* + * Process a BGWRITER message. */ static void pgstat_recv_bgwriter(PgStat_MsgBgWriter *msg, int len) @@ -4122,11 +3946,8 @@ pgstat_recv_bgwriter(PgStat_MsgBgWriter *msg, int len) globalStats.bgwriter.buf_alloc += msg->m_buf_alloc; } -/* ---------- - * pgstat_recv_checkpointer() - - * - * Process a CHECKPOINTER message. - * ---------- +/* + * Process a CHECKPOINTER message. */ static void pgstat_recv_checkpointer(PgStat_MsgCheckpointer *msg, int len) @@ -4140,11 +3961,8 @@ pgstat_recv_checkpointer(PgStat_MsgCheckpointer *msg, int len) globalStats.checkpointer.buf_fsync_backend += msg->m_buf_fsync_backend; } -/* ---------- - * pgstat_recv_wal() - - * - * Process a WAL message. - * ---------- +/* + * Process a WAL message. */ static void pgstat_recv_wal(PgStat_MsgWal *msg, int len) @@ -4159,11 +3977,8 @@ pgstat_recv_wal(PgStat_MsgWal *msg, int len) walStats.wal_sync_time += msg->m_wal_sync_time; } -/* ---------- - * pgstat_recv_slru() - - * - * Process a SLRU message. - * ---------- +/* + * Process a SLRU message. */ static void pgstat_recv_slru(PgStat_MsgSLRU *msg, int len) @@ -4177,11 +3992,8 @@ pgstat_recv_slru(PgStat_MsgSLRU *msg, int len) slruStats[msg->m_index].truncate += msg->m_truncate; } -/* ---------- - * pgstat_recv_recoveryconflict() - - * - * Process a RECOVERYCONFLICT message. - * ---------- +/* + * Process a RECOVERYCONFLICT message. */ static void pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int len) @@ -4217,11 +4029,8 @@ pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int len) } } -/* ---------- - * pgstat_recv_deadlock() - - * - * Process a DEADLOCK message. - * ---------- +/* + * Process a DEADLOCK message. */ static void pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len) @@ -4233,11 +4042,8 @@ pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len) dbentry->n_deadlocks++; } -/* ---------- - * pgstat_recv_checksum_failure() - - * - * Process a CHECKSUMFAILURE message. - * ---------- +/* + * Process a CHECKSUMFAILURE message. */ static void pgstat_recv_checksum_failure(PgStat_MsgChecksumFailure *msg, int len) @@ -4250,11 +4056,8 @@ pgstat_recv_checksum_failure(PgStat_MsgChecksumFailure *msg, int len) dbentry->last_checksum_failure = msg->m_failure_time; } -/* ---------- - * pgstat_recv_replslot() - - * - * Process a REPLSLOT message. - * ---------- +/* + * Process a REPLSLOT message. */ static void pgstat_recv_replslot(PgStat_MsgReplSlot *msg, int len) @@ -4301,11 +4104,8 @@ pgstat_recv_replslot(PgStat_MsgReplSlot *msg, int len) } } -/* ---------- - * pgstat_recv_connect() - - * - * Process a CONNECT message. - * ---------- +/* + * Process a CONNECT message. */ static void pgstat_recv_connect(PgStat_MsgConnect *msg, int len) @@ -4316,11 +4116,8 @@ pgstat_recv_connect(PgStat_MsgConnect *msg, int len) dbentry->n_sessions++; } -/* ---------- - * pgstat_recv_disconnect() - - * - * Process a DISCONNECT message. - * ---------- +/* + * Process a DISCONNECT message. */ static void pgstat_recv_disconnect(PgStat_MsgDisconnect *msg, int len) @@ -4347,11 +4144,8 @@ pgstat_recv_disconnect(PgStat_MsgDisconnect *msg, int len) } } -/* ---------- - * pgstat_recv_tempfile() - - * - * Process a TEMPFILE message. - * ---------- +/* + * Process a TEMPFILE message. */ static void pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len) @@ -4364,11 +4158,8 @@ pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len) dbentry->n_temp_files += 1; } -/* ---------- - * pgstat_recv_funcstat() - - * - * Count what the backend has done. - * ---------- +/* + * Count what the backend has done. */ static void pgstat_recv_funcstat(PgStat_MsgFuncstat *msg, int len) @@ -4412,11 +4203,8 @@ pgstat_recv_funcstat(PgStat_MsgFuncstat *msg, int len) } } -/* ---------- - * pgstat_recv_funcpurge() - - * - * Arrange for dead function removal. - * ---------- +/* + * Arrange for dead function removal. */ static void pgstat_recv_funcpurge(PgStat_MsgFuncpurge *msg, int len) @@ -4444,11 +4232,8 @@ pgstat_recv_funcpurge(PgStat_MsgFuncpurge *msg, int len) } } -/* ---------- - * pgstat_recv_subscription_drop() - - * - * Process a SUBSCRIPTIONDROP message. - * ---------- +/* + * Process a SUBSCRIPTIONDROP message. */ static void pgstat_recv_subscription_drop(PgStat_MsgSubscriptionDrop *msg, int len) @@ -4462,11 +4247,8 @@ pgstat_recv_subscription_drop(PgStat_MsgSubscriptionDrop *msg, int len) HASH_REMOVE, NULL); } -/* ---------- - * pgstat_recv_subscription_error() - - * - * Process a SUBSCRIPTIONERROR message. - * ---------- +/* + * Process a SUBSCRIPTIONERROR message. */ static void pgstat_recv_subscription_error(PgStat_MsgSubscriptionError *msg, int len) diff --git a/src/backend/utils/activity/pgstat_archiver.c b/src/backend/utils/activity/pgstat_archiver.c index 36788f7ab44..4a37b8f6e76 100644 --- a/src/backend/utils/activity/pgstat_archiver.c +++ b/src/backend/utils/activity/pgstat_archiver.c @@ -21,12 +21,9 @@ #include "utils/timestamp.h" -/* ---------- - * pgstat_send_archiver() - - * - * Tell the collector about the WAL file that we successfully - * archived or failed to archive. - * ---------- +/* + * Tell the collector about the WAL file that we successfully + * archived or failed to archive. */ void pgstat_send_archiver(const char *xlog, bool failed) diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index c5cf0002de6..0d9421666d8 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -28,11 +28,8 @@ PgStat_MsgBgWriter PendingBgWriterStats; -/* ---------- - * pgstat_send_bgwriter() - - * - * Send bgwriter statistics to the collector - * ---------- +/* + * Send bgwriter statistics to the collector */ void pgstat_send_bgwriter(void) diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 2ce3fba76c7..f8358eb4e0c 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -28,11 +28,8 @@ PgStat_MsgCheckpointer PendingCheckpointerStats; -/* ---------- - * pgstat_send_checkpointer() - - * - * Send checkpointer statistics to the collector - * ---------- +/* + * Send checkpointer statistics to the collector */ void pgstat_send_checkpointer(void) diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 31f6624c588..b4fba80835a 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -36,13 +36,10 @@ SessionEndType pgStatSessionEndCause = DISCONNECT_NORMAL; static PgStat_Counter pgLastSessionReportTime = 0; -/* ---------- - * pgstat_drop_database() - - * - * Tell the collector that we just dropped a database. - * (If the message gets lost, we will still clean the dead DB eventually - * via future invocations of pgstat_vacuum_stat().) - * ---------- +/* + * Tell the collector that we just dropped a database. + * (If the message gets lost, we will still clean the dead DB eventually + * via future invocations of pgstat_vacuum_stat().) */ void pgstat_drop_database(Oid databaseid) @@ -57,11 +54,8 @@ pgstat_drop_database(Oid databaseid) pgstat_send(&msg, sizeof(msg)); } -/* -------- - * pgstat_report_recovery_conflict() - - * - * Tell the collector about a Hot Standby recovery conflict. - * -------- +/* + * Tell the collector about a Hot Standby recovery conflict. */ void pgstat_report_recovery_conflict(int reason) @@ -77,11 +71,8 @@ pgstat_report_recovery_conflict(int reason) pgstat_send(&msg, sizeof(msg)); } -/* -------- - * pgstat_report_deadlock() - - * - * Tell the collector about a deadlock detected. - * -------- +/* + * Tell the collector about a deadlock detected. */ void pgstat_report_deadlock(void) @@ -96,11 +87,8 @@ pgstat_report_deadlock(void) pgstat_send(&msg, sizeof(msg)); } -/* -------- - * pgstat_report_checksum_failures_in_db() - - * - * Tell the collector about one or more checksum failures. - * -------- +/* + * Tell the collector about one or more checksum failures. */ void pgstat_report_checksum_failures_in_db(Oid dboid, int failurecount) @@ -118,11 +106,8 @@ pgstat_report_checksum_failures_in_db(Oid dboid, int failurecount) pgstat_send(&msg, sizeof(msg)); } -/* -------- - * pgstat_report_checksum_failure() - - * - * Tell the collector about a checksum failure. - * -------- +/* + * Tell the collector about a checksum failure. */ void pgstat_report_checksum_failure(void) @@ -130,11 +115,8 @@ pgstat_report_checksum_failure(void) pgstat_report_checksum_failures_in_db(MyDatabaseId, 1); } -/* -------- - * pgstat_report_tempfile() - - * - * Tell the collector about a temporary file. - * -------- +/* + * Tell the collector about a temporary file. */ void pgstat_report_tempfile(size_t filesize) @@ -150,11 +132,8 @@ pgstat_report_tempfile(size_t filesize) pgstat_send(&msg, sizeof(msg)); } -/* -------- - * pgstat_report_connect() - - * - * Tell the collector about a new connection. - * -------- +/* + * Tell the collector about a new connection. */ void pgstat_report_connect(Oid dboid) @@ -171,11 +150,8 @@ pgstat_report_connect(Oid dboid) pgstat_send(&msg, sizeof(PgStat_MsgConnect)); } -/* -------- - * pgstat_report_disconnect() - - * - * Tell the collector about a disconnect. - * -------- +/* + * Tell the collector about a disconnect. */ void pgstat_report_disconnect(Oid dboid) @@ -262,15 +238,12 @@ pgstat_update_dbstats(PgStat_MsgTabstat *tsmsg, TimestampTz now) } } -/* -------- - * pgstat_should_report_connstats() - - * - * We report session statistics only for normal backend processes. Parallel - * workers run in parallel, so they don't contribute to session times, even - * though they use CPU time. Walsender processes could be considered here, - * but they have different session characteristics from normal backends (for - * example, they are always "active"), so they would skew session statistics. - * ---------- +/* + * We report session statistics only for normal backend processes. Parallel + * workers run in parallel, so they don't contribute to session times, even + * though they use CPU time. Walsender processes could be considered here, + * but they have different session characteristics from normal backends (for + * example, they are always "active"), so they would skew session statistics. */ static bool pgstat_should_report_connstat(void) diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index ad37bb74aa6..93ec29757aa 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -206,8 +206,7 @@ pgstat_send_funcstats(void) } /* - * find_funcstat_entry - find any existing PgStat_BackendFunctionEntry entry - * for specified function + * find any existing PgStat_BackendFunctionEntry entry for specified function * * If no entry, return NULL, don't create a new one */ diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index f5c03b54b48..74b62c39c0d 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -95,17 +95,14 @@ bool have_relation_stats; static HTAB *pgStatTabHash = NULL; -/* ---------- - * pgstat_relation_init() - +/* + * Initialize a relcache entry to count access statistics. + * Called whenever a relation is opened. * - * Initialize a relcache entry to count access statistics. - * Called whenever a relation is opened. - * - * We assume that a relcache entry's pgstat_info field is zeroed by - * relcache.c when the relcache entry is made; thereafter it is long-lived - * data. We can avoid repeated searches of the TabStatus arrays when the - * same relation is touched repeatedly within a transaction. - * ---------- + * We assume that a relcache entry's pgstat_info field is zeroed by + * relcache.c when the relcache entry is made; thereafter it is long-lived + * data. We can avoid repeated searches of the TabStatus arrays when the + * same relation is touched repeatedly within a transaction. */ void pgstat_relation_init(Relation rel) @@ -141,16 +138,13 @@ pgstat_relation_init(Relation rel) rel->pgstat_info = get_tabstat_entry(rel_id, rel->rd_rel->relisshared); } -/* ---------- - * pgstat_drop_relation() - +/* + * Tell the collector that we just dropped a relation. + * (If the message gets lost, we will still clean the dead entry eventually + * via future invocations of pgstat_vacuum_stat().) * - * Tell the collector that we just dropped a relation. - * (If the message gets lost, we will still clean the dead entry eventually - * via future invocations of pgstat_vacuum_stat().) - * - * Currently not used for lack of any good place to call it; we rely - * entirely on pgstat_vacuum_stat() to clean out stats for dead rels. - * ---------- + * Currently not used for lack of any good place to call it; we rely + * entirely on pgstat_vacuum_stat() to clean out stats for dead rels. */ #ifdef NOT_USED void @@ -173,13 +167,10 @@ pgstat_drop_relation(Oid relid) } #endif /* NOT_USED */ -/* ---------- - * pgstat_report_autovac() - - * - * Called from autovacuum.c to report startup of an autovacuum process. - * We are called before InitPostgres is done, so can't rely on MyDatabaseId; - * the db OID must be passed in, instead. - * ---------- +/* + * Called from autovacuum.c to report startup of an autovacuum process. + * We are called before InitPostgres is done, so can't rely on MyDatabaseId; + * the db OID must be passed in, instead. */ void pgstat_report_autovac(Oid dboid) @@ -196,11 +187,8 @@ pgstat_report_autovac(Oid dboid) pgstat_send(&msg, sizeof(msg)); } -/* --------- - * pgstat_report_vacuum() - - * - * Tell the collector about the table we just vacuumed. - * --------- +/* + * Tell the collector about the table we just vacuumed. */ void pgstat_report_vacuum(Oid tableoid, bool shared, @@ -221,14 +209,11 @@ pgstat_report_vacuum(Oid tableoid, bool shared, pgstat_send(&msg, sizeof(msg)); } -/* -------- - * pgstat_report_analyze() - - * - * Tell the collector about the table we just analyzed. +/* + * Tell the collector about the table we just analyzed. * * Caller must provide new live- and dead-tuples estimates, as well as a * flag indicating whether to reset the changes_since_analyze counter. - * -------- */ void pgstat_report_analyze(Relation rel, @@ -281,7 +266,7 @@ pgstat_report_analyze(Relation rel, } /* - * pgstat_count_heap_insert - count a tuple insertion of n tuples + * count a tuple insertion of n tuples */ void pgstat_count_heap_insert(Relation rel, PgStat_Counter n) @@ -296,7 +281,7 @@ pgstat_count_heap_insert(Relation rel, PgStat_Counter n) } /* - * pgstat_count_heap_update - count a tuple update + * count a tuple update */ void pgstat_count_heap_update(Relation rel, bool hot) @@ -315,7 +300,7 @@ pgstat_count_heap_update(Relation rel, bool hot) } /* - * pgstat_count_heap_delete - count a tuple deletion + * count a tuple deletion */ void pgstat_count_heap_delete(Relation rel) @@ -330,7 +315,7 @@ pgstat_count_heap_delete(Relation rel) } /* - * pgstat_count_truncate - update tuple counters due to truncate + * update tuple counters due to truncate */ void pgstat_count_truncate(Relation rel) @@ -348,7 +333,7 @@ pgstat_count_truncate(Relation rel) } /* - * pgstat_update_heap_dead_tuples - update dead-tuples count + * update dead-tuples count * * The semantics of this are that we are reporting the nontransactional * recovery of "delta" dead tuples; so t_delta_dead_tuples decreases @@ -367,7 +352,7 @@ pgstat_update_heap_dead_tuples(Relation rel, int delta) } /* - * find_tabstat_entry - find any existing PgStat_TableStatus entry for rel + * find any existing PgStat_TableStatus entry for rel * * If no entry, return NULL, don't create a new one * @@ -772,7 +757,7 @@ pgstat_send_tabstat(PgStat_MsgTabstat *tsmsg, TimestampTz now) } /* - * get_tabstat_entry - find or create a PgStat_TableStatus entry for rel + * find or create a PgStat_TableStatus entry for rel */ static PgStat_TableStatus * get_tabstat_entry(Oid rel_id, bool isshared) @@ -858,7 +843,7 @@ get_tabstat_entry(Oid rel_id, bool isshared) } /* - * add_tabstat_xact_level - add a new (sub)transaction state record + * add a new (sub)transaction state record */ static void add_tabstat_xact_level(PgStat_TableStatus *pgstat_info, int nest_level) @@ -898,8 +883,6 @@ ensure_tabstat_xact_level(PgStat_TableStatus *pgstat_info) } /* - * pgstat_truncdrop_save_counters - * * Whenever a table is truncated/dropped, we save its i/u/d counters so that * they can be cleared, and if the (sub)xact that executed the truncate/drop * later aborts, the counters can be restored to the saved (pre-truncate/drop) @@ -921,7 +904,7 @@ pgstat_truncdrop_save_counters(PgStat_TableXactStatus *trans, bool is_drop) } /* - * pgstat_truncdrop_restore_counters - restore counters when a truncate aborts + * restore counters when a truncate aborts */ static void pgstat_truncdrop_restore_counters(PgStat_TableXactStatus *trans) diff --git a/src/backend/utils/activity/pgstat_replslot.c b/src/backend/utils/activity/pgstat_replslot.c index 2d575b6e5c1..a9405cd135e 100644 --- a/src/backend/utils/activity/pgstat_replslot.c +++ b/src/backend/utils/activity/pgstat_replslot.c @@ -22,15 +22,12 @@ #include "utils/pgstat_internal.h" -/* ---------- - * pgstat_reset_replslot_counter() - +/* + * Tell the statistics collector to reset a single replication slot + * counter, or all replication slots counters (when name is null). * - * Tell the statistics collector to reset a single replication slot - * counter, or all replication slots counters (when name is null). - * - * Permission checking for this function is managed through the normal - * GRANT system. - * ---------- + * Permission checking for this function is managed through the normal + * GRANT system. */ void pgstat_reset_replslot_counter(const char *name) @@ -53,11 +50,8 @@ pgstat_reset_replslot_counter(const char *name) pgstat_send(&msg, sizeof(msg)); } -/* ---------- - * pgstat_report_replslot() - - * - * Tell the collector about replication slot statistics. - * ---------- +/* + * Tell the collector about replication slot statistics. */ void pgstat_report_replslot(const PgStat_StatReplSlotEntry *repSlotStat) @@ -82,11 +76,8 @@ pgstat_report_replslot(const PgStat_StatReplSlotEntry *repSlotStat) pgstat_send(&msg, sizeof(PgStat_MsgReplSlot)); } -/* ---------- - * pgstat_report_replslot_create() - - * - * Tell the collector about creating the replication slot. - * ---------- +/* + * Tell the collector about creating the replication slot. */ void pgstat_report_replslot_create(const char *slotname) @@ -100,11 +91,8 @@ pgstat_report_replslot_create(const char *slotname) pgstat_send(&msg, sizeof(PgStat_MsgReplSlot)); } -/* ---------- - * pgstat_report_replslot_drop() - - * - * Tell the collector about dropping the replication slot. - * ---------- +/* + * Tell the collector about dropping the replication slot. */ void pgstat_report_replslot_drop(const char *slotname) diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 058a926211f..3e284900b15 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -32,15 +32,12 @@ static inline PgStat_MsgSLRU *slru_entry(int slru_idx); static PgStat_MsgSLRU SLRUStats[SLRU_NUM_ELEMENTS]; -/* ---------- - * pgstat_reset_slru_counter() - +/* + * Tell the statistics collector to reset a single SLRU counter, or all + * SLRU counters (when name is null). * - * Tell the statistics collector to reset a single SLRU counter, or all - * SLRU counters (when name is null). - * - * Permission checking for this function is managed through the normal - * GRANT system. - * ---------- + * Permission checking for this function is managed through the normal + * GRANT system. */ void pgstat_reset_slru_counter(const char *name) @@ -103,8 +100,6 @@ pgstat_count_slru_truncate(int slru_idx) } /* - * pgstat_slru_name - * * Returns SLRU name for an index. The index may be above SLRU_NUM_ELEMENTS, * in which case this returns NULL. This allows writing code that does not * know the number of entries in advance. @@ -119,8 +114,6 @@ pgstat_slru_name(int slru_idx) } /* - * pgstat_slru_index - * * Determine index of entry for a SLRU with a given name. If there's no exact * match, returns index of the last "other" entry used for SLRUs defined in * external projects. @@ -140,11 +133,8 @@ pgstat_slru_index(const char *name) return (SLRU_NUM_ELEMENTS - 1); } -/* ---------- - * pgstat_send_slru() - - * - * Send SLRU statistics to the collector - * ---------- +/* + * Send SLRU statistics to the collector */ void pgstat_send_slru(void) @@ -179,8 +169,6 @@ pgstat_send_slru(void) } /* - * slru_entry - * * Returns pointer to entry with counters for given SLRU (based on the name * stored in SlruCtl as lwlock tranche name). */ diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 2f1168f5e40..70874e13f5e 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -20,15 +20,12 @@ #include "utils/pgstat_internal.h" -/* ---------- - * pgstat_reset_subscription_counter() - +/* + * Tell the statistics collector to reset a single subscription + * counter, or all subscription counters (when subid is InvalidOid). * - * Tell the statistics collector to reset a single subscription - * counter, or all subscription counters (when subid is InvalidOid). - * - * Permission checking for this function is managed through the normal - * GRANT system. - * ---------- + * Permission checking for this function is managed through the normal + * GRANT system. */ void pgstat_reset_subscription_counter(Oid subid) @@ -44,11 +41,8 @@ pgstat_reset_subscription_counter(Oid subid) pgstat_send(&msg, sizeof(msg)); } -/* ---------- - * pgstat_report_subscription_error() - - * - * Tell the collector about the subscription error. - * ---------- +/* + * Tell the collector about the subscription error. */ void pgstat_report_subscription_error(Oid subid, bool is_apply_error) @@ -61,11 +55,8 @@ pgstat_report_subscription_error(Oid subid, bool is_apply_error) pgstat_send(&msg, sizeof(PgStat_MsgSubscriptionError)); } -/* ---------- - * pgstat_report_subscription_drop() - - * - * Tell the collector about dropping the subscription. - * ---------- +/* + * Tell the collector about dropping the subscription. */ void pgstat_report_subscription_drop(Oid subid) diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 14b6394033c..ad999887630 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -38,14 +38,11 @@ PgStat_MsgWal WalStats; static WalUsage prevWalUsage; -/* ---------- - * pgstat_send_wal() - - * - * Send WAL statistics to the collector. +/* + * Send WAL statistics to the collector. * * If 'force' is not set, WAL stats message is only sent if enough time has * passed since last one was sent to reach PGSTAT_STAT_INTERVAL. - * ---------- */ void pgstat_send_wal(bool force) -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0002-pgstat-remove-some-superflous-comments-from-pgst.patch" From 88f36d242b874c2e5523a04ae025478b7a1b6860 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Mon, 21 Mar 2022 12:21:26 -0700 Subject: [PATCH v68 02/31] pgstat: remove some superflous comments from pgstat.h. These would all need to be rephrased, but since they don't add value, remove them instead. The comments for PgStat_Msg* are left in, because they will all be removed in the near future. Author: Andres Freund Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- src/include/pgstat.h | 36 ------------------------------------ 1 file changed, 36 deletions(-) diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 534d595ca08..3584078f6ea 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -776,9 +776,6 @@ typedef union PgStat_Msg #define PGSTAT_FILE_FORMAT_ID 0x01A5BCA6 -/* - * Archiver statistics kept in the stats collector - */ typedef struct PgStat_ArchiverStats { PgStat_Counter archived_count; /* archival successes */ @@ -792,9 +789,6 @@ typedef struct PgStat_ArchiverStats TimestampTz stat_reset_timestamp; } PgStat_ArchiverStats; -/* - * Background writer statistics kept in the stats collector - */ typedef struct PgStat_BgWriterStats { PgStat_Counter buf_written_clean; @@ -803,9 +797,6 @@ typedef struct PgStat_BgWriterStats TimestampTz stat_reset_timestamp; } PgStat_BgWriterStats; -/* - * Checkpointer statistics kept in the stats collector - */ typedef struct PgStat_CheckpointerStats { TimestampTz stats_timestamp; /* time of stats file update */ @@ -818,10 +809,6 @@ typedef struct PgStat_CheckpointerStats PgStat_Counter buf_fsync_backend; } PgStat_CheckpointerStats; -/* ---------- - * PgStat_StatDBEntry The collector's data per database - * ---------- - */ typedef struct PgStat_StatDBEntry { Oid databaseid; @@ -866,10 +853,6 @@ typedef struct PgStat_StatDBEntry HTAB *functions; } PgStat_StatDBEntry; -/* ---------- - * PgStat_StatFuncEntry The collector's data per function - * ---------- - */ typedef struct PgStat_StatFuncEntry { Oid functionid; @@ -880,9 +863,6 @@ typedef struct PgStat_StatFuncEntry PgStat_Counter f_self_time; } PgStat_StatFuncEntry; -/* - * Global statistics kept in the stats collector - */ typedef struct PgStat_GlobalStats { TimestampTz stats_timestamp; /* time of stats file update */ @@ -891,9 +871,6 @@ typedef struct PgStat_GlobalStats PgStat_BgWriterStats bgwriter; } PgStat_GlobalStats; -/* - * Replication slot statistics kept in the stats collector - */ typedef struct PgStat_StatReplSlotEntry { NameData slotname; @@ -908,9 +885,6 @@ typedef struct PgStat_StatReplSlotEntry TimestampTz stat_reset_timestamp; } PgStat_StatReplSlotEntry; -/* - * SLRU statistics kept in the stats collector - */ typedef struct PgStat_SLRUStats { PgStat_Counter blocks_zeroed; @@ -923,9 +897,6 @@ typedef struct PgStat_SLRUStats TimestampTz stat_reset_timestamp; } PgStat_SLRUStats; -/* - * Subscription statistics kept in the stats collector. - */ typedef struct PgStat_StatSubEntry { Oid subid; /* hash key (must be first) */ @@ -935,10 +906,6 @@ typedef struct PgStat_StatSubEntry TimestampTz stat_reset_timestamp; } PgStat_StatSubEntry; -/* ---------- - * PgStat_StatTabEntry The collector's data per table (or index) - * ---------- - */ typedef struct PgStat_StatTabEntry { Oid tableid; @@ -971,9 +938,6 @@ typedef struct PgStat_StatTabEntry PgStat_Counter autovac_analyze_count; } PgStat_StatTabEntry; -/* - * WAL statistics kept in the stats collector - */ typedef struct PgStat_WalStats { PgStat_Counter wal_records; -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0003-dshash-revise-sequential-scan-support.patch" From 86e6f7336312b683659941f31c13b2a0e59b68bd Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Wed, 23 Mar 2022 16:38:10 -0700 Subject: [PATCH v68 03/31] dshash: revise sequential scan support. The previous coding of dshash_seq_next(), on the first call, accessed status->hash_table->size_log2 without holding a partition lock and without guaranteeing that ensure_valid_bucket_pointers() had ever been called. That oversight turns out to not have immediately visible effects, because bucket 0 is always in partition 0, and ensure_valid_bucket_pointers() was called after acquiring the partition lock. However, PARTITION_FOR_BUCKET_INDEX() with a size_log2 of 0 ends up triggering formally undefined behaviour. Simplify by accessing partition 0, without using PARTITION_FOR_BUCKET_INDEX(). While at it, remove dshash_get_current(), there is no convincing use case. Also polish a few comments. --- src/include/lib/dshash.h | 1 - src/backend/lib/dshash.c | 58 +++++++++++++++++++--------------------- 2 files changed, 28 insertions(+), 31 deletions(-) diff --git a/src/include/lib/dshash.h b/src/include/lib/dshash.h index caeb60ad723..28f8db2eabc 100644 --- a/src/include/lib/dshash.h +++ b/src/include/lib/dshash.h @@ -101,7 +101,6 @@ extern void dshash_seq_init(dshash_seq_status *status, dshash_table *hash_table, extern void *dshash_seq_next(dshash_seq_status *status); extern void dshash_seq_term(dshash_seq_status *status); extern void dshash_delete_current(dshash_seq_status *status); -extern void *dshash_get_current(dshash_seq_status *status); /* Convenience hash and compare functions wrapping memcmp and tag_hash. */ extern int dshash_memcmp(const void *a, const void *b, size_t size, void *arg); diff --git a/src/backend/lib/dshash.c b/src/backend/lib/dshash.c index 84a9db47c77..5f03e1c5aae 100644 --- a/src/backend/lib/dshash.c +++ b/src/backend/lib/dshash.c @@ -601,13 +601,12 @@ dshash_memhash(const void *v, size_t size, void *arg) } /* - * dshash_seq_init/_next/_term - * Sequentially scan through dshash table and return all the - * elements one by one, return NULL when no more. + * Sequentially scan through dshash table and return all the elements one by + * one, return NULL when no more. * - * dshash_seq_term should always be called when a scan finished. - * The caller may delete returned elements midst of a scan by using - * dshash_delete_current(). exclusive must be true to delete elements. + * dshash_seq_term needs to be called when a scan finished. The caller may + * delete returned elements midst of a scan by using dshash_delete_current() + * if exclusive = true. */ void dshash_seq_init(dshash_seq_status *status, dshash_table *hash_table, @@ -625,34 +624,38 @@ dshash_seq_init(dshash_seq_status *status, dshash_table *hash_table, /* * Returns the next element. * - * Returned elements are locked and the caller must not explicitly release - * it. It is released at the next call to dshash_next(). + * Returned elements are locked and the caller may not release the lock. It is + * released by future calls to dshash_seq_next() or dshash_seq_term(). */ void * dshash_seq_next(dshash_seq_status *status) { dsa_pointer next_item_pointer; - if (status->curitem == NULL) + /* + * Not yet holding any partition locks. Need to determine the size of the + * hash table, it could have been resized since we were looking + * last. Since we iterate in partition order, we can start by + * unconditionally lock partition 0. + * + * Once we hold the lock, no resizing can happen until the scan ends. So + * we don't need to repeatedly call ensure_valid_bucket_pointers(). + */ + if (status->curpartition == -1) { - int partition; - Assert(status->curbucket == 0); Assert(!status->hash_table->find_locked); - /* first shot. grab the first item. */ - partition = - PARTITION_FOR_BUCKET_INDEX(status->curbucket, - status->hash_table->size_log2); - LWLockAcquire(PARTITION_LOCK(status->hash_table, partition), - status->exclusive ? LW_EXCLUSIVE : LW_SHARED); - status->curpartition = partition; + status->curpartition = 0; + + LWLockAcquire(PARTITION_LOCK(status->hash_table, + status->curpartition), + status->exclusive ? LW_EXCLUSIVE : LW_SHARED); - /* resize doesn't happen from now until seq scan ends */ - status->nbuckets = - NUM_BUCKETS(status->hash_table->control->size_log2); ensure_valid_bucket_pointers(status->hash_table); + status->nbuckets = + NUM_BUCKETS(status->hash_table->control->size_log2); next_item_pointer = status->hash_table->buckets[status->curbucket]; } else @@ -714,7 +717,7 @@ dshash_seq_next(dshash_seq_status *status) /* * Terminates the seqscan and release all locks. * - * Should be always called when finishing or exiting a seqscan. + * Needs to be called after finishing or when exiting a seqscan. */ void dshash_seq_term(dshash_seq_status *status) @@ -726,7 +729,9 @@ dshash_seq_term(dshash_seq_status *status) LWLockRelease(PARTITION_LOCK(status->hash_table, status->curpartition)); } -/* Remove the current entry while a seq scan. */ +/* + * Remove the current entry of the seq scan. + */ void dshash_delete_current(dshash_seq_status *status) { @@ -746,13 +751,6 @@ dshash_delete_current(dshash_seq_status *status) delete_item(hash_table, item); } -/* Get the current entry while a seq scan. */ -void * -dshash_get_current(dshash_seq_status *status) -{ - return ENTRY_FROM_ITEM(status->curitem); -} - /* * Print debugging information about the internal state of the hash table to * stderr. The caller must hold no partition locks. -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0004-dsm-allow-use-in-single-user-mode.patch" From 219d2fea28e689389cc98c1e8242bd9ecada6318 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Mon, 28 Mar 2022 20:49:14 -0700 Subject: [PATCH v68 04/31] dsm: allow use in single user mode. This is mostly useful for the shared memory stats, to avoid a dedicated code path for single user mode. --- src/backend/storage/ipc/dsm.c | 9 +++++++-- src/backend/storage/ipc/dsm_impl.c | 4 +++- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c index e9e9fae3eb9..00a8c08ea39 100644 --- a/src/backend/storage/ipc/dsm.c +++ b/src/backend/storage/ipc/dsm.c @@ -397,6 +397,7 @@ static void dsm_backend_startup(void) { #ifdef EXEC_BACKEND + if (IsUnderPostmaster) { void *control_address = NULL; @@ -496,8 +497,12 @@ dsm_create(Size size, int flags) FreePageManager *dsm_main_space_fpm = dsm_main_space_begin; bool using_main_dsm_region = false; - /* Unsafe in postmaster (and pointless in a stand-alone backend). */ - Assert(IsUnderPostmaster); + /* + * Unsafe in postmaster. Might seem pointless to allow in single user + * mode, but this way subsystems do not need dedicated single user mode + * paths. + */ + Assert(IsUnderPostmaster || !IsPostmasterEnvironment); if (!dsm_init_done) dsm_backend_startup(); diff --git a/src/backend/storage/ipc/dsm_impl.c b/src/backend/storage/ipc/dsm_impl.c index 49f4c98620c..6ec865612f1 100644 --- a/src/backend/storage/ipc/dsm_impl.c +++ b/src/backend/storage/ipc/dsm_impl.c @@ -959,6 +959,7 @@ dsm_impl_pin_segment(dsm_handle handle, void *impl_private, { #ifdef USE_DSM_WINDOWS case DSM_IMPL_WINDOWS: + if (IsUnderPostmaster) { HANDLE hmap; @@ -984,8 +985,8 @@ dsm_impl_pin_segment(dsm_handle handle, void *impl_private, * is unpinned, dsm_impl_unpin_segment can close it. */ *impl_private_pm_handle = hmap; - break; } + break; #endif default: break; @@ -1008,6 +1009,7 @@ dsm_impl_unpin_segment(dsm_handle handle, void **impl_private) { #ifdef USE_DSM_WINDOWS case DSM_IMPL_WINDOWS: + if (IsUnderPostmaster) { if (*impl_private && !DuplicateHandle(PostmasterHandle, *impl_private, -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0005-pgstat-stats-collector-references-in-comments.patch" From a7253368e46604d77f426dfa4f0d04e5ab9ac59e Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Tue, 29 Mar 2022 20:34:49 -0700 Subject: [PATCH v68 05/31] pgstat: stats collector references in comments Author: Andres Freund Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- src/include/pgstat.h | 4 +-- src/include/utils/backend_status.h | 4 +-- src/include/utils/pgstat_internal.h | 2 +- src/backend/access/heap/heapam_handler.c | 4 +-- src/backend/access/heap/vacuumlazy.c | 8 +++--- src/backend/commands/analyze.c | 2 +- src/backend/commands/dbcommands.c | 2 +- src/backend/commands/matview.c | 8 +++--- src/backend/commands/subscriptioncmds.c | 4 +-- src/backend/commands/vacuum.c | 4 +-- src/backend/postmaster/autovacuum.c | 14 +++++----- src/backend/postmaster/bgwriter.c | 4 +-- src/backend/postmaster/checkpointer.c | 14 +++------- src/backend/postmaster/pgarch.c | 8 +++--- src/backend/postmaster/pgstat.c | 28 +++++++++---------- src/backend/postmaster/walwriter.c | 9 +++--- src/backend/replication/basebackup.c | 4 +-- src/backend/tcop/postgres.c | 8 +++--- src/backend/utils/activity/pgstat_archiver.c | 3 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 14 +++++----- src/backend/utils/activity/pgstat_relation.c | 16 +++++------ src/backend/utils/activity/pgstat_replslot.c | 10 +++---- src/backend/utils/activity/pgstat_slru.c | 3 +- .../utils/activity/pgstat_subscription.c | 8 +++--- src/backend/utils/adt/pgstatfuncs.c | 4 +-- src/backend/utils/error/elog.c | 2 +- contrib/test_decoding/t/001_repl_stats.pl | 3 +- src/test/regress/expected/stats.out | 2 +- src/test/regress/sql/stats.sql | 2 +- 31 files changed, 95 insertions(+), 107 deletions(-) diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 3584078f6ea..7ce26f61642 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -1,7 +1,7 @@ /* ---------- * pgstat.h * - * Definitions for the PostgreSQL statistics collector daemon. + * Definitions for the PostgreSQL cumulative statistics system. * * Copyright (c) 2001-2022, PostgreSQL Global Development Group * @@ -86,7 +86,7 @@ typedef enum PgStat_Single_Reset_Type * it against zeroes to detect whether there are any counts to transmit. * * Note that the time counters are in instr_time format here. We convert to - * microseconds in PgStat_Counter format when transmitting to the collector. + * microseconds in PgStat_Counter format when flushing out pending statistics. * ---------- */ typedef struct PgStat_FunctionCounts diff --git a/src/include/utils/backend_status.h b/src/include/utils/backend_status.h index 8217d0cb6b7..7403bca25ed 100644 --- a/src/include/utils/backend_status.h +++ b/src/include/utils/backend_status.h @@ -85,8 +85,8 @@ typedef struct PgBackendGSSStatus * * Each live backend maintains a PgBackendStatus struct in shared memory * showing its current activity. (The structs are allocated according to - * BackendId, but that is not critical.) Note that the collector process - * has no involvement in, or even access to, these structs. + * BackendId, but that is not critical.) Note that this is unrelated to the + * cumulative stats system (i.e. pgstat.c et al). * * Each auxiliary process also maintains a PgBackendStatus struct in shared * memory. diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index abbb4f8d964..b95e1f56eab 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -1,7 +1,7 @@ /* ---------- * pgstat_internal.h * - * Definitions for the PostgreSQL activity statistics facility that should + * Definitions for the PostgreSQL cumulative statistics system that should * only be needed by files implementing statistics support (rather than ones * reporting / querying stats). * diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index dee264e8596..d16685ca15c 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1089,8 +1089,8 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, * our own. In this case we should count and sample the row, * to accommodate users who load a table and analyze it in one * transaction. (pgstat_report_analyze has to adjust the - * numbers we send to the stats collector to make this come - * out right.) + * numbers we report to the cumulative stats system to make + * this come out right.) */ if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetXmin(targtuple->t_data))) { diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c index 9b7114c1591..3ef43868a63 100644 --- a/src/backend/access/heap/vacuumlazy.c +++ b/src/backend/access/heap/vacuumlazy.c @@ -592,9 +592,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, &frozenxid_updated, &minmulti_updated, false); /* - * Report results to the stats collector, too. + * Report results to the cumulative stats system, too. * - * Deliberately avoid telling the stats collector about LP_DEAD items that + * Deliberately avoid telling the stats system about LP_DEAD items that * remain in the table due to VACUUM bypassing index and heap vacuuming. * ANALYZE will consider the remaining LP_DEAD items to be dead "tuples". * It seems like a good idea to err on the side of not vacuuming again too @@ -2240,10 +2240,10 @@ lazy_vacuum(LVRelState *vacrel) * dead_items space is not CPU cache resident. * * We don't take any special steps to remember the LP_DEAD items (such - * as counting them in our final report to the stats collector) when + * as counting them in our final update to the stats system) when * the optimization is applied. Though the accounting used in * analyze.c's acquire_sample_rows() will recognize the same LP_DEAD - * items as dead rows in its own stats collector report, that's okay. + * items as dead rows in its own stats report, that's okay. * The discrepancy should be negligible. If this optimization is ever * expanded to cover more cases then this may need to be reconsidered. */ diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c index 736479295ad..305226692a4 100644 --- a/src/backend/commands/analyze.c +++ b/src/backend/commands/analyze.c @@ -680,7 +680,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params, } /* - * Now report ANALYZE to the stats collector. For regular tables, we do + * Now report ANALYZE to the cumulative stats system. For regular tables, we do * it only if not doing inherited stats. For partitioned tables, we only * do it for inherited stats. (We're never called for not-inherited stats * on partitioned tables anyway.) diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c index df16533901e..ce776c53cad 100644 --- a/src/backend/commands/dbcommands.c +++ b/src/backend/commands/dbcommands.c @@ -1665,7 +1665,7 @@ dropdb(const char *dbname, bool missing_ok, bool force) DropDatabaseBuffers(db_id); /* - * Tell the stats collector to forget it immediately, too. + * Tell the cumulative stats system to forget it immediately, too. */ pgstat_drop_database(db_id); diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c index 05e7b60059d..9ab248d25e0 100644 --- a/src/backend/commands/matview.c +++ b/src/backend/commands/matview.c @@ -338,10 +338,10 @@ ExecRefreshMatView(RefreshMatViewStmt *stmt, const char *queryString, refresh_by_heap_swap(matviewOid, OIDNewHeap, relpersistence); /* - * Inform stats collector about our activity: basically, we truncated - * the matview and inserted some new data. (The concurrent code path - * above doesn't need to worry about this because the inserts and - * deletes it issues get counted by lower-level code.) + * Inform cumulative stats system about our activity: basically, we + * truncated the matview and inserted some new data. (The concurrent + * code path above doesn't need to worry about this because the inserts + * and deletes it issues get counted by lower-level code.) */ pgstat_count_truncate(matviewRel); if (!stmt->skipData) diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 85dacbe93d6..51505373ea4 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -1409,7 +1409,7 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel) * slot stays dropped even if the transaction rolls back. So we cannot * run DROP SUBSCRIPTION inside a transaction block if dropping the * replication slot. Also, in this case, we report a message for dropping - * the subscription to the stats collector. + * the subscription to the cumulative stats system. * * XXX The command name should really be something like "DROP SUBSCRIPTION * of a subscription that is associated with a replication slot", but we @@ -1583,7 +1583,7 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel) PG_END_TRY(); /* - * Send a message for dropping this subscription to the stats collector. + * Tell the cumulative stats system that the subscription is getting dropped. * We can safely report dropping the subscription statistics here if the * subscription is associated with a replication slot since we cannot run * DROP SUBSCRIPTION inside a transaction block. Subscription statistics diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c index deec4887bec..f5c92c3860e 100644 --- a/src/backend/commands/vacuum.c +++ b/src/backend/commands/vacuum.c @@ -352,8 +352,8 @@ vacuum(List *relations, VacuumParams *params, errmsg("PROCESS_TOAST required with VACUUM FULL"))); /* - * Send info about dead objects to the statistics collector, unless we are - * in autovacuum --- autovacuum.c does this for itself. + * Send info about dead objects to the cumulative stats system, unless + * we are in autovacuum --- autovacuum.c does this for itself. */ if ((params->options & VACOPT_VACUUM) && !IsAutoVacuumWorkerProcess()) pgstat_vacuum_stat(); diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c index 681ef91b81e..c6d30fa5271 100644 --- a/src/backend/postmaster/autovacuum.c +++ b/src/backend/postmaster/autovacuum.c @@ -1689,7 +1689,7 @@ AutoVacWorkerMain(int argc, char *argv[]) char dbname[NAMEDATALEN]; /* - * Report autovac startup to the stats collector. We deliberately do + * Report autovac startup to the cumulative stats system. We deliberately do * this before InitPostgres, so that the last_autovac_time will get * updated even if the connection attempt fails. This is to prevent * autovac from getting "stuck" repeatedly selecting an unopenable @@ -1996,9 +1996,9 @@ do_autovacuum(void) StartTransactionCommand(); /* - * Clean up any dead statistics collector entries for this DB. We always - * want to do this exactly once per DB-processing cycle, even if we find - * nothing worth vacuuming in the database. + * Clean up any dead statistics entries for this DB. We always want to do + * this exactly once per DB-processing cycle, even if we find nothing + * worth vacuuming in the database. */ pgstat_vacuum_stat(); @@ -3041,7 +3041,7 @@ recheck_relation_needs_vacanalyze(Oid relid, * * For analyze, the analysis done is that the number of tuples inserted, * deleted and updated since the last analyze exceeds a threshold calculated - * in the same fashion as above. Note that the collector actually stores + * in the same fashion as above. Note that the cumulative stats system stores * the number of tuples (both live and dead) that there were as of the last * analyze. This is asymmetric to the VACUUM case. * @@ -3051,8 +3051,8 @@ recheck_relation_needs_vacanalyze(Oid relid, * * A table whose autovacuum_enabled option is false is * automatically skipped (unless we have to vacuum it due to freeze_max_age). - * Thus autovacuum can be disabled for specific tables. Also, when the stats - * collector does not have data about a table, it will be skipped. + * Thus autovacuum can be disabled for specific tables. Also, when the cumulative + * stats system does not have data about a table, it will be skipped. * * A table whose vac_base_thresh value is < 0 takes the base value from the * autovacuum_vacuum_threshold GUC variable. Similarly, a vac_scale_factor diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index d1f5d12eff8..447596b2a8b 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -240,9 +240,7 @@ BackgroundWriterMain(void) */ can_hibernate = BgBufferSync(&wb_context); - /* - * Send off activity statistics to the stats collector - */ + /* Report pending statistics to the cumulative stats system */ pgstat_send_bgwriter(); if (FirstCallSinceLastCheckpoint()) diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index a59c3cf0201..80914aa891c 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -492,12 +492,8 @@ CheckpointerMain(void) /* Check for archive_timeout and switch xlog files if necessary. */ CheckArchiveTimeout(); - /* - * Send off activity statistics to the stats collector. - */ + /* Report pending statistics to the cumulative stats system */ pgstat_send_checkpointer(); - - /* Send WAL statistics to the stats collector. */ pgstat_send_wal(true); /* @@ -570,8 +566,8 @@ HandleCheckpointerInterrupts(void) * Close down the database. * * Since ShutdownXLOG() creates restartpoint or checkpoint, and - * updates the statistics, increment the checkpoint request and send - * the statistics to the stats collector. + * updates the statistics, increment the checkpoint request and flush + * out pending statistic. */ PendingCheckpointerStats.m_requested_checkpoints++; ShutdownXLOG(0, 0); @@ -718,9 +714,7 @@ CheckpointWriteDelay(int flags, double progress) CheckArchiveTimeout(); - /* - * Report interim activity statistics. - */ + /* Report interim statistics to the cumulative stats system */ pgstat_send_checkpointer(); /* diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c index d916ed39a8c..adfa404f464 100644 --- a/src/backend/postmaster/pgarch.c +++ b/src/backend/postmaster/pgarch.c @@ -477,8 +477,8 @@ pgarch_ArchiverCopyLoop(void) pgarch_archiveDone(xlog); /* - * Tell the collector about the WAL file that we successfully - * archived + * Tell the cumulative stats system about the WAL file that we + * successfully archived */ pgstat_send_archiver(xlog, false); @@ -487,8 +487,8 @@ pgarch_ArchiverCopyLoop(void) else { /* - * Tell the collector about the WAL file that we failed to - * archive + * Tell the cumulative stats system about the WAL file that we + * failed to archive */ pgstat_send_archiver(xlog, true); diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index ef1cba61a6f..de981a89a5b 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/postmaster/pgstat.c @@ -695,9 +695,9 @@ allow_immediate_pgstat_restart(void) /* * Shut down a single backend's statistics reporting at process exit. * - * Flush any remaining statistics counts out to the collector. - * Without this, operations triggered during backend exit (such as - * temp table deletions) won't be counted. + * Flush out any remaining statistics counts. Without this, operations + * triggered during backend exit (such as temp table deletions) won't be + * counted. */ static void pgstat_shutdown_hook(int code, Datum arg) @@ -705,10 +705,10 @@ pgstat_shutdown_hook(int code, Datum arg) Assert(!pgstat_is_shutdown); /* - * If we got as far as discovering our own database ID, we can report what - * we did to the collector. Otherwise, we'd be sending an invalid - * database ID, so forget it. (This means that accesses to pg_database - * during failed backend starts might never get counted.) + * If we got as far as discovering our own database ID, we can flush out + * what we did so far. Otherwise, we'd be reporting an invalid database + * ID, so forget it. (This means that accesses to pg_database during + * failed backend starts might never get counted.) */ if (OidIsValid(MyDatabaseId)) pgstat_report_stat(true); @@ -1219,7 +1219,7 @@ pgstat_collect_oids(Oid catalogid, AttrNumber anum_oid) } /* - * Tell the statistics collector to reset counters for our database. + * Reset counters for our database. * * Permission checking for this function is managed through the normal * GRANT system. @@ -1238,7 +1238,7 @@ pgstat_reset_counters(void) } /* - * Tell the statistics collector to reset a single counter. + * Reset a single counter. * * Permission checking for this function is managed through the normal * GRANT system. @@ -1260,7 +1260,7 @@ pgstat_reset_single_counter(Oid objoid, PgStat_Single_Reset_Type type) } /* - * Tell the statistics collector to reset cluster-wide shared counters. + * Reset cluster-wide shared counters. * * Permission checking for this function is managed through the normal * GRANT system. @@ -1322,8 +1322,8 @@ pgstat_send_inquiry(TimestampTz clock_time, TimestampTz cutoff_time, Oid databas /* * Support function for the SQL-callable pgstat* functions. Returns * the collected statistics for one database or NULL. NULL doesn't mean - * that the database doesn't exist, it is just not yet known by the - * collector, so the caller is better off to report ZERO instead. + * that the database doesn't exist, just that there are no statistics, so the + * caller is better off to report ZERO instead. */ PgStat_StatDBEntry * pgstat_fetch_stat_dbentry(Oid dbid) @@ -1357,8 +1357,8 @@ pgstat_fetch_global(void) /* * Support function for the SQL-callable pgstat* functions. Returns * the collected statistics for one table or NULL. NULL doesn't mean - * that the table doesn't exist, it is just not yet known by the - * collector, so the caller is better off to report ZERO instead. + * that the table doesn't exist, just that there are no statistics, so the + * caller is better off to report ZERO instead. */ PgStat_StatTabEntry * pgstat_fetch_stat_tabentry(Oid relid) diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 102fa2a089f..e2c7eb78d69 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -257,7 +257,7 @@ WalWriterMain(void) else if (left_till_hibernate > 0) left_till_hibernate--; - /* Send WAL statistics to the stats collector */ + /* report pending statistics to the cumulative stats system */ pgstat_send_wal(false); /* @@ -295,12 +295,11 @@ HandleWalWriterInterrupts(void) if (ShutdownRequestPending) { /* - * Force to send remaining WAL statistics to the stats collector at - * process exit. + * Force reporting remaining WAL statistics at process exit. * * Since pgstat_send_wal is invoked with 'force' is false in main loop - * to avoid overloading to the stats collector, there may exist unsent - * stats counters for the WAL writer. + * to avoid overloading the cumulative stats system, there may exist + * unreported stats counters for the WAL writer. */ pgstat_send_wal(true); diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c index 6884cad2c00..db9fd0a7290 100644 --- a/src/backend/replication/basebackup.c +++ b/src/backend/replication/basebackup.c @@ -1507,8 +1507,8 @@ is_checksummed_file(const char *fullpath, const char *filename) * * If 'missing_ok' is true, will not throw an error if the file is not found. * - * If dboid is anything other than InvalidOid then any checksum failures detected - * will get reported to the stats collector. + * If dboid is anything other than InvalidOid then any checksum failures + * detected will get reported to the cumulative stats system. * * Returns true if the file was successfully sent, false if 'missing_ok', * and the file did not exist. diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index ba2fcfeb4af..260b650f15f 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -2937,7 +2937,7 @@ die(SIGNAL_ARGS) ProcDiePending = true; } - /* for the statistics collector */ + /* for the cumulative stats system */ pgStatSessionEndCause = DISCONNECT_KILLED; /* If we're still here, waken anything waiting on the process latch */ @@ -4368,8 +4368,8 @@ PostgresMain(const char *dbname, const char *username) * * Note: this includes fflush()'ing the last of the prior output. * - * This is also a good time to send collected statistics to the - * collector, and to update the PS stats display. We avoid doing + * This is also a good time to flush out collected statistics to the + * cumulative stats system, and to update the PS stats display. We avoid doing * those every time through the message loop because it'd slow down * processing of batched messages, and because we don't want to report * uncommitted updates (that confuses autovacuum). The notification @@ -4710,7 +4710,7 @@ PostgresMain(const char *dbname, const char *username) */ case EOF: - /* for the statistics collector */ + /* for the cumulative statistics system */ pgStatSessionEndCause = DISCONNECT_CLIENT_EOF; /* FALLTHROUGH */ diff --git a/src/backend/utils/activity/pgstat_archiver.c b/src/backend/utils/activity/pgstat_archiver.c index 4a37b8f6e76..c829b91b60c 100644 --- a/src/backend/utils/activity/pgstat_archiver.c +++ b/src/backend/utils/activity/pgstat_archiver.c @@ -22,8 +22,7 @@ /* - * Tell the collector about the WAL file that we successfully - * archived or failed to archive. + * Report archiver statistics */ void pgstat_send_archiver(const char *xlog, bool failed) diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index 0d9421666d8..7e7eae80372 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -29,7 +29,7 @@ PgStat_MsgBgWriter PendingBgWriterStats; /* - * Send bgwriter statistics to the collector + * Report bgwriter statistics */ void pgstat_send_bgwriter(void) diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index f8358eb4e0c..dbb9d861a4b 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -29,7 +29,7 @@ PgStat_MsgCheckpointer PendingCheckpointerStats; /* - * Send checkpointer statistics to the collector + * Report checkpointer statistics */ void pgstat_send_checkpointer(void) diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index b4fba80835a..0f2dd956d36 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -55,7 +55,7 @@ pgstat_drop_database(Oid databaseid) } /* - * Tell the collector about a Hot Standby recovery conflict. + * Report a Hot Standby recovery conflict. */ void pgstat_report_recovery_conflict(int reason) @@ -72,7 +72,7 @@ pgstat_report_recovery_conflict(int reason) } /* - * Tell the collector about a deadlock detected. + * Report a detected deadlock. */ void pgstat_report_deadlock(void) @@ -88,7 +88,7 @@ pgstat_report_deadlock(void) } /* - * Tell the collector about one or more checksum failures. + * Report one or more checksum failures. */ void pgstat_report_checksum_failures_in_db(Oid dboid, int failurecount) @@ -107,7 +107,7 @@ pgstat_report_checksum_failures_in_db(Oid dboid, int failurecount) } /* - * Tell the collector about a checksum failure. + * Report one checksum failure in the current database. */ void pgstat_report_checksum_failure(void) @@ -116,7 +116,7 @@ pgstat_report_checksum_failure(void) } /* - * Tell the collector about a temporary file. + * Report creation of temporary file. */ void pgstat_report_tempfile(size_t filesize) @@ -133,7 +133,7 @@ pgstat_report_tempfile(size_t filesize) } /* - * Tell the collector about a new connection. + * Notify stats system of a new connection. */ void pgstat_report_connect(Oid dboid) @@ -151,7 +151,7 @@ pgstat_report_connect(Oid dboid) } /* - * Tell the collector about a disconnect. + * Notify the stats system of a disconnect. */ void pgstat_report_disconnect(Oid dboid) diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index 74b62c39c0d..6413b3cc132 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -188,7 +188,7 @@ pgstat_report_autovac(Oid dboid) } /* - * Tell the collector about the table we just vacuumed. + * Report that the table was just vacuumed. */ void pgstat_report_vacuum(Oid tableoid, bool shared, @@ -210,7 +210,7 @@ pgstat_report_vacuum(Oid tableoid, bool shared, } /* - * Tell the collector about the table we just analyzed. + * Report that the table was just analyzed. * * Caller must provide new live- and dead-tuples estimates, as well as a * flag indicating whether to reset the changes_since_analyze counter. @@ -230,10 +230,10 @@ pgstat_report_analyze(Relation rel, * already inserted and/or deleted rows in the target table. ANALYZE will * have counted such rows as live or dead respectively. Because we will * report our counts of such rows at transaction end, we should subtract - * off these counts from what we send to the collector now, else they'll - * be double-counted after commit. (This approach also ensures that the - * collector ends up with the right numbers if we abort instead of - * committing.) + * off these counts from what is already written to shared stats now, else + * they'll be double-counted after commit. (This approach also ensures + * that the shared stats ends up with the right numbers if we abort + * instead of committing.) * * Waste no time on partitioned tables, though. */ @@ -557,8 +557,8 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state) /* * All we need do here is unlink the transaction stats state from the * nontransactional state. The nontransactional action counts will be - * reported to the stats collector immediately, while the effects on - * live and dead tuple counts are preserved in the 2PC state file. + * reported to the stats system immediately, while the effects on live and + * dead tuple counts are preserved in the 2PC state file. * * Note: AtEOXact_PgStat_Relations is not called during PREPARE. */ diff --git a/src/backend/utils/activity/pgstat_replslot.c b/src/backend/utils/activity/pgstat_replslot.c index a9405cd135e..35078ad73c8 100644 --- a/src/backend/utils/activity/pgstat_replslot.c +++ b/src/backend/utils/activity/pgstat_replslot.c @@ -23,8 +23,8 @@ /* - * Tell the statistics collector to reset a single replication slot - * counter, or all replication slots counters (when name is null). + * Reset counters for a single replication slot, or all replication slots + * (when name is null). * * Permission checking for this function is managed through the normal * GRANT system. @@ -51,7 +51,7 @@ pgstat_reset_replslot_counter(const char *name) } /* - * Tell the collector about replication slot statistics. + * Report replication slot statistics. */ void pgstat_report_replslot(const PgStat_StatReplSlotEntry *repSlotStat) @@ -77,7 +77,7 @@ pgstat_report_replslot(const PgStat_StatReplSlotEntry *repSlotStat) } /* - * Tell the collector about creating the replication slot. + * Report replication slot creation. */ void pgstat_report_replslot_create(const char *slotname) @@ -92,7 +92,7 @@ pgstat_report_replslot_create(const char *slotname) } /* - * Tell the collector about dropping the replication slot. + * Report replication slot drop. */ void pgstat_report_replslot_drop(const char *slotname) diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 3e284900b15..6dae3a5dc66 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -33,8 +33,7 @@ static PgStat_MsgSLRU SLRUStats[SLRU_NUM_ELEMENTS]; /* - * Tell the statistics collector to reset a single SLRU counter, or all - * SLRU counters (when name is null). + * Reset counters for a single SLRU, or all SLRUs (when name is null). * * Permission checking for this function is managed through the normal * GRANT system. diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 70874e13f5e..2ee23d5ae2c 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -21,8 +21,8 @@ /* - * Tell the statistics collector to reset a single subscription - * counter, or all subscription counters (when subid is InvalidOid). + * Reset counters for a single subscription, or all subscriptions (when subid + * is InvalidOid). * * Permission checking for this function is managed through the normal * GRANT system. @@ -42,7 +42,7 @@ pgstat_reset_subscription_counter(Oid subid) } /* - * Tell the collector about the subscription error. + * Report a subscription error. */ void pgstat_report_subscription_error(Oid subid, bool is_apply_error) @@ -56,7 +56,7 @@ pgstat_report_subscription_error(Oid subid, bool is_apply_error) } /* - * Tell the collector about dropping the subscription. + * Report dropping the subscription. */ void pgstat_report_subscription_drop(Oid subid) diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index ce84525d402..fd4276fbc67 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -1,7 +1,7 @@ /*------------------------------------------------------------------------- * * pgstatfuncs.c - * Functions for accessing the statistics collector data + * Functions for accessing various forms of statistics data * * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California @@ -1820,7 +1820,7 @@ pg_stat_get_slru(PG_FUNCTION_ARGS) SetSingleFuncCall(fcinfo, 0); - /* request SLRU stats from the stat collector */ + /* request SLRU stats from the cumulative stats system */ stats = pgstat_fetch_slru(); for (i = 0;; i++) diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c index 7402696986b..bd4b2c19b1c 100644 --- a/src/backend/utils/error/elog.c +++ b/src/backend/utils/error/elog.c @@ -647,7 +647,7 @@ errfinish(const char *filename, int lineno, const char *funcname) fflush(stderr); /* - * Let the statistics collector know. Only mark the session as + * Let the cumulative stats system know. Only mark the session as * terminated by fatal error if there is no other known cause. */ if (pgStatSessionEndCause == DISCONNECT_NORMAL) diff --git a/contrib/test_decoding/t/001_repl_stats.pl b/contrib/test_decoding/t/001_repl_stats.pl index fd82244a15c..eda8897703d 100644 --- a/contrib/test_decoding/t/001_repl_stats.pl +++ b/contrib/test_decoding/t/001_repl_stats.pl @@ -88,8 +88,7 @@ regression_slot3|t|t), # Test to remove one of the replication slots and adjust # max_replication_slots accordingly to the number of slots. This leads # to a mismatch between the number of slots present in the stats file and the -# number of stats present in the shared memory, simulating the scenario for -# drop slot message lost by the statistics collector process. We verify +# number of stats present in the shared memory. We verify # replication statistics data is fine after restart. $node->stop; diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out index 06a1d2f229b..57550d49e77 100644 --- a/src/test/regress/expected/stats.out +++ b/src/test/regress/expected/stats.out @@ -1,5 +1,5 @@ -- --- Test Statistics Collector +-- Test cumulative stats subsystem -- -- Must be run after tenk2 has been created (by create_table), -- populated (by create_misc) and indexed (by create_index). diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql index ae1ec173e32..7e746dca3b7 100644 --- a/src/test/regress/sql/stats.sql +++ b/src/test/regress/sql/stats.sql @@ -1,5 +1,5 @@ -- --- Test Statistics Collector +-- Test cumulative stats subsystem -- -- Must be run after tenk2 has been created (by create_table), -- populated (by create_misc) and indexed (by create_index). -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0006-pgstat-add-pgstat_copy_relation_stats.patch" From eea5d4b5dd12e66097f911590c76564353c9e161 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Tue, 29 Mar 2022 21:42:36 -0700 Subject: [PATCH v68 06/31] pgstat: add pgstat_copy_relation_stats(). Author: Andres Freund Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- src/include/pgstat.h | 2 ++ src/backend/catalog/index.c | 26 ++-------------- src/backend/utils/activity/pgstat_relation.c | 32 ++++++++++++++++++++ 3 files changed, 36 insertions(+), 24 deletions(-) diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 7ce26f61642..ed37523ce17 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -1058,6 +1058,8 @@ extern PgStat_BackendFunctionEntry *find_funcstat_entry(Oid func_id); * Functions in pgstat_relation.c */ +extern void pgstat_copy_relation_stats(Relation dstrel, Relation srcrel); + extern void pgstat_relation_init(Relation rel); extern void pgstat_report_autovac(Oid dboid); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index dd715ca0609..063be25062b 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1734,30 +1734,8 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName) changeDependenciesOf(RelationRelationId, oldIndexId, newIndexId); changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId); - /* - * Copy over statistics from old to new index - */ - { - PgStat_StatTabEntry *tabentry; - - tabentry = pgstat_fetch_stat_tabentry(oldIndexId); - if (tabentry) - { - if (pgstat_relation_should_count(newClassRel)) - { - newClassRel->pgstat_info->t_counts.t_numscans = tabentry->numscans; - newClassRel->pgstat_info->t_counts.t_tuples_returned = tabentry->tuples_returned; - newClassRel->pgstat_info->t_counts.t_tuples_fetched = tabentry->tuples_fetched; - newClassRel->pgstat_info->t_counts.t_blocks_fetched = tabentry->blocks_fetched; - newClassRel->pgstat_info->t_counts.t_blocks_hit = tabentry->blocks_hit; - - /* - * The data will be sent by the next pgstat_report_stat() - * call. - */ - } - } - } + /* copy over statistics from old to new index */ + pgstat_copy_relation_stats(newClassRel, oldClassRel); /* Copy data of pg_statistic from the old index to the new one */ CopyStatistics(oldIndexId, newIndexId); diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index 6413b3cc132..b0ac406e5fd 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -95,6 +95,38 @@ bool have_relation_stats; static HTAB *pgStatTabHash = NULL; +/* + * Copy stats between relations. This is used for things like REINDEX + * CONCURRENTLY. + */ +void +pgstat_copy_relation_stats(Relation dst, Relation src) +{ + PgStat_StatTabEntry *srcstats; + + srcstats = pgstat_fetch_stat_tabentry(RelationGetRelid(src)); + + if (!srcstats) + return; + + if (pgstat_relation_should_count(dst)) + { + /* + * XXX: temporarily this does not actually quite do what the name + * says, and just copy index related fields. A subsequent commit will + * do more. + */ + + dst->pgstat_info->t_counts.t_numscans = srcstats->numscans; + dst->pgstat_info->t_counts.t_tuples_returned = srcstats->tuples_returned; + dst->pgstat_info->t_counts.t_tuples_fetched = srcstats->tuples_fetched; + dst->pgstat_info->t_counts.t_blocks_fetched = srcstats->blocks_fetched; + dst->pgstat_info->t_counts.t_blocks_hit = srcstats->blocks_hit; + + /* the data will be sent by the next pgstat_report_stat() call */ + } +} + /* * Initialize a relcache entry to count access statistics. * Called whenever a relation is opened. -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0007-pgstat-move-transactional-code-into-pgstat_xact..patch" From 0163480f31887ce6ce581b951ddc115885e7fc6c Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Tue, 29 Mar 2022 21:57:22 -0700 Subject: [PATCH v68 07/31] pgstat: move transactional code into pgstat_xact.c. Author: Reviewed-By: Discussion: https://postgr.es/m/ Backpatch: --- src/include/utils/pgstat_internal.h | 8 +- src/backend/postmaster/pgstat.c | 184 ++++------------------- src/backend/utils/activity/Makefile | 3 +- src/backend/utils/activity/pgstat_xact.c | 139 +++++++++++++++++ 4 files changed, 178 insertions(+), 156 deletions(-) create mode 100644 src/backend/utils/activity/pgstat_xact.c diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index b95e1f56eab..7f4d79470f7 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -77,7 +77,6 @@ static const char *const slru_names[] = { * Functions in pgstat.c */ -extern PgStat_SubXactStatus *pgstat_xact_stack_level_get(int nest_level); extern void pgstat_setheader(PgStat_MsgHdr *hdr, StatMsgType mtype); extern void pgstat_send(void *msg, int len); #ifdef USE_ASSERT_CHECKING @@ -129,6 +128,13 @@ extern void pgstat_wal_initialize(void); extern bool pgstat_wal_pending(void); + +/* + * Functions in pgstat_xact.c + */ + +extern PgStat_SubXactStatus *pgstat_xact_stack_level_get(int nest_level); + /* * Variables in pgstat.c */ diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index de981a89a5b..8cbed1d1bc3 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/postmaster/pgstat.c @@ -198,8 +198,6 @@ static time_t last_pgstat_start_time; static bool pgStatRunningInCollector = false; -static PgStat_SubXactStatus *pgStatXactStack = NULL; - /* * Info about current "snapshot" of stats file */ @@ -740,158 +738,6 @@ pgstat_initialize(void) } -/* ------------------------------------------------------------ - * Transaction integration - * ------------------------------------------------------------ - */ - -/* - * Called from access/transam/xact.c at top-level transaction commit/abort. - */ -void -AtEOXact_PgStat(bool isCommit, bool parallel) -{ - PgStat_SubXactStatus *xact_state; - - AtEOXact_PgStat_Database(isCommit, parallel); - - /* handle transactional stats information */ - xact_state = pgStatXactStack; - if (xact_state != NULL) - { - Assert(xact_state->nest_level == 1); - Assert(xact_state->prev == NULL); - - AtEOXact_PgStat_Relations(xact_state, isCommit); - } - pgStatXactStack = NULL; - - /* Make sure any stats snapshot is thrown away */ - pgstat_clear_snapshot(); -} - -/* - * Called from access/transam/xact.c at subtransaction commit/abort. - */ -void -AtEOSubXact_PgStat(bool isCommit, int nestDepth) -{ - PgStat_SubXactStatus *xact_state; - - /* merge the sub-transaction's transactional stats into the parent */ - xact_state = pgStatXactStack; - if (xact_state != NULL && - xact_state->nest_level >= nestDepth) - { - /* delink xact_state from stack immediately to simplify reuse case */ - pgStatXactStack = xact_state->prev; - - AtEOSubXact_PgStat_Relations(xact_state, isCommit, nestDepth); - - pfree(xact_state); - } -} - -/* - * Save the transactional stats state at 2PC transaction prepare. - */ -void -AtPrepare_PgStat(void) -{ - PgStat_SubXactStatus *xact_state; - - xact_state = pgStatXactStack; - if (xact_state != NULL) - { - Assert(xact_state->nest_level == 1); - Assert(xact_state->prev == NULL); - - AtPrepare_PgStat_Relations(xact_state); - } -} - -/* - * Clean up after successful PREPARE. - * - * Note: AtEOXact_PgStat is not called during PREPARE. - */ -void -PostPrepare_PgStat(void) -{ - PgStat_SubXactStatus *xact_state; - - /* - * We don't bother to free any of the transactional state, since it's all - * in TopTransactionContext and will go away anyway. - */ - xact_state = pgStatXactStack; - if (xact_state != NULL) - { - Assert(xact_state->nest_level == 1); - Assert(xact_state->prev == NULL); - - PostPrepare_PgStat_Relations(xact_state); - } - pgStatXactStack = NULL; - - /* Make sure any stats snapshot is thrown away */ - pgstat_clear_snapshot(); -} - -/* - * Discard any data collected in the current transaction. Any subsequent - * request will cause new snapshots to be read. - * - * This is also invoked during transaction commit or abort to discard - * the no-longer-wanted snapshot. - */ -void -pgstat_clear_snapshot(void) -{ - pgstat_assert_is_up(); - - /* Release memory, if any was allocated */ - if (pgStatLocalContext) - MemoryContextDelete(pgStatLocalContext); - - /* Reset variables */ - pgStatLocalContext = NULL; - pgStatDBHash = NULL; - replSlotStatHash = NULL; - subscriptionStatHash = NULL; - - /* - * Historically the backend_status.c facilities lived in this file, and - * were reset with the same function. For now keep it that way, and - * forward the reset request. - */ - pgstat_clear_backend_activity_snapshot(); -} - -/* - * Ensure (sub)transaction stack entry for the given nest_level exists, adding - * it if needed. - */ -PgStat_SubXactStatus * -pgstat_xact_stack_level_get(int nest_level) -{ - PgStat_SubXactStatus *xact_state; - - xact_state = pgStatXactStack; - if (xact_state == NULL || xact_state->nest_level != nest_level) - { - xact_state = (PgStat_SubXactStatus *) - MemoryContextAlloc(TopTransactionContext, - sizeof(PgStat_SubXactStatus)); - xact_state->nest_level = nest_level; - xact_state->prev = pgStatXactStack; - xact_state->first = NULL; - pgStatXactStack = xact_state; - } - return xact_state; -} - - /* ------------------------------------------------------------ * Public functions used by backends follow * ------------------------------------------------------------ @@ -1319,6 +1165,36 @@ pgstat_send_inquiry(TimestampTz clock_time, TimestampTz cutoff_time, Oid databas pgstat_send(&msg, sizeof(msg)); } +/* + * Discard any data collected in the current transaction. Any subsequent + * request will cause new snapshots to be read. + * + * This is also invoked during transaction commit or abort to discard + * the no-longer-wanted snapshot. + */ +void +pgstat_clear_snapshot(void) +{ + pgstat_assert_is_up(); + + /* Release memory, if any was allocated */ + if (pgStatLocalContext) + MemoryContextDelete(pgStatLocalContext); + + /* Reset variables */ + pgStatLocalContext = NULL; + pgStatDBHash = NULL; + replSlotStatHash = NULL; + subscriptionStatHash = NULL; + + /* + * Historically the backend_status.c facilities lived in this file, and + * were reset with the same function. For now keep it that way, and + * forward the reset request. + */ + pgstat_clear_backend_activity_snapshot(); +} + /* * Support function for the SQL-callable pgstat* functions. Returns * the collected statistics for one database or NULL. NULL doesn't mean diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile index 25a967ab7d7..791ba68e7e3 100644 --- a/src/backend/utils/activity/Makefile +++ b/src/backend/utils/activity/Makefile @@ -23,9 +23,10 @@ OBJS = \ pgstat_function.o \ pgstat_relation.o \ pgstat_replslot.o \ + pgstat_slru.o \ pgstat_subscription.o \ pgstat_wal.o \ - pgstat_slru.o \ + pgstat_xact.o \ wait_event.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c new file mode 100644 index 00000000000..17907e32789 --- /dev/null +++ b/src/backend/utils/activity/pgstat_xact.c @@ -0,0 +1,139 @@ +/* ------------------------------------------------------------------------- + * + * pgstat_xact.c + * Transactional integration for the cumulative statistics system. + * + * Copyright (c) 2001-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/utils/activity/pgstat_xact.c + * ------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/transam.h" +#include "access/xact.h" +#include "pgstat.h" +#include "utils/memutils.h" +#include "utils/pgstat_internal.h" + + +static PgStat_SubXactStatus *pgStatXactStack = NULL; + + +/* + * Called from access/transam/xact.c at top-level transaction commit/abort. + */ +void +AtEOXact_PgStat(bool isCommit, bool parallel) +{ + PgStat_SubXactStatus *xact_state; + + AtEOXact_PgStat_Database(isCommit, parallel); + + /* handle transactional stats information */ + xact_state = pgStatXactStack; + if (xact_state != NULL) + { + Assert(xact_state->nest_level == 1); + Assert(xact_state->prev == NULL); + + AtEOXact_PgStat_Relations(xact_state, isCommit); + } + pgStatXactStack = NULL; + + /* Make sure any stats snapshot is thrown away */ + pgstat_clear_snapshot(); +} + +/* + * Called from access/transam/xact.c at subtransaction commit/abort. + */ +void +AtEOSubXact_PgStat(bool isCommit, int nestDepth) +{ + PgStat_SubXactStatus *xact_state; + + /* merge the sub-transaction's transactional stats into the parent */ + xact_state = pgStatXactStack; + if (xact_state != NULL && + xact_state->nest_level >= nestDepth) + { + /* delink xact_state from stack immediately to simplify reuse case */ + pgStatXactStack = xact_state->prev; + + AtEOSubXact_PgStat_Relations(xact_state, isCommit, nestDepth); + + pfree(xact_state); + } +} + +/* + * Save the transactional stats state at 2PC transaction prepare. + */ +void +AtPrepare_PgStat(void) +{ + PgStat_SubXactStatus *xact_state; + + xact_state = pgStatXactStack; + if (xact_state != NULL) + { + Assert(xact_state->nest_level == 1); + Assert(xact_state->prev == NULL); + + AtPrepare_PgStat_Relations(xact_state); + } +} + +/* + * Clean up after successful PREPARE. + * + * Note: AtEOXact_PgStat is not called during PREPARE. + */ +void +PostPrepare_PgStat(void) +{ + PgStat_SubXactStatus *xact_state; + + /* + * We don't bother to free any of the transactional state, since it's all + * in TopTransactionContext and will go away anyway. + */ + xact_state = pgStatXactStack; + if (xact_state != NULL) + { + Assert(xact_state->nest_level == 1); + Assert(xact_state->prev == NULL); + + PostPrepare_PgStat_Relations(xact_state); + } + pgStatXactStack = NULL; + + /* Make sure any stats snapshot is thrown away */ + pgstat_clear_snapshot(); +} + +/* + * Ensure (sub)transaction stack entry for the given nest_level exists, adding + * it if needed. + */ +PgStat_SubXactStatus * +pgstat_xact_stack_level_get(int nest_level) +{ + PgStat_SubXactStatus *xact_state; + + xact_state = pgStatXactStack; + if (xact_state == NULL || xact_state->nest_level != nest_level) + { + xact_state = (PgStat_SubXactStatus *) + MemoryContextAlloc(TopTransactionContext, + sizeof(PgStat_SubXactStatus)); + xact_state->nest_level = nest_level; + xact_state->prev = pgStatXactStack; + xact_state->first = NULL; + pgStatXactStack = xact_state; + } + return xact_state; +} -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0008-pgstat-introduce-PgStat_Kind-enum.patch" From cc4422002ad7e991db42a09710f68eff04980972 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Fri, 1 Apr 2022 13:43:49 -0700 Subject: [PATCH v68 08/31] pgstat: introduce PgStat_Kind enum. Will be used by following commits to generalize stats infrastructure. --- src/include/pgstat.h | 27 +++++++++++++++++++++++++++ src/tools/pgindent/typedefs.list | 1 + 2 files changed, 28 insertions(+) diff --git a/src/include/pgstat.h b/src/include/pgstat.h index ed37523ce17..1d03bd97de4 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -33,6 +33,33 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* The types of statistics entries */ +typedef enum PgStat_Kind +{ + /* + * Don't define an INVALID value so switch() statements can warn if some + * cases aren't covered. But define the first member to 1 so that + * uninitialized values can be detected more easily. + */ + + /* stats with a variable number of entries */ + PGSTAT_KIND_DATABASE = 1, /* database-wide statistics */ + PGSTAT_KIND_RELATION, /* per-table statistics */ + PGSTAT_KIND_FUNCTION, /* per-function statistics */ + PGSTAT_KIND_REPLSLOT, /* per-slot statistics */ + PGSTAT_KIND_SUBSCRIPTION, /* per-subscription statistics */ + + /* stats with a constant number of entries */ + PGSTAT_KIND_ARCHIVER, + PGSTAT_KIND_BGWRITER, + PGSTAT_KIND_CHECKPOINTER, + PGSTAT_KIND_SLRU, + PGSTAT_KIND_WAL, +} PgStat_Kind; + +#define PGSTAT_KIND_FIRST PGSTAT_KIND_DATABASE +#define PGSTAT_KIND_LAST PGSTAT_KIND_WAL +#define PGSTAT_NUM_KINDS (PGSTAT_KIND_LAST + 1) /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 72fafb795bb..a5c15a1d68f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1931,6 +1931,7 @@ PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_FunctionEntry PgStat_GlobalStats +PgStat_Kind PgStat_Msg PgStat_MsgAnalyze PgStat_MsgAnlAncestors -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0009-pgstat-prepare-APIs-used-by-pgstatfuncs-for-shar.patch" From 8679b23153d3b663a3a720060362ad82ffd7b45f Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sun, 3 Apr 2022 12:57:31 -0700 Subject: [PATCH v68 09/31] pgstat: prepare APIs used by pgstatfuncs for shared memory stats. --- src/include/pgstat.h | 25 ++----- src/backend/postmaster/pgstat.c | 45 +++++++------ src/backend/utils/activity/pgstat_replslot.c | 51 +++++++++++--- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 20 +++++- src/backend/utils/adt/pgstatfuncs.c | 67 +++++++++---------- src/tools/pgindent/typedefs.list | 2 - 7 files changed, 123 insertions(+), 93 deletions(-) diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 1d03bd97de4..a87b45f368e 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -85,21 +85,6 @@ typedef enum SessionEndType */ typedef int64 PgStat_Counter; -/* Possible targets for resetting cluster-wide shared values */ -typedef enum PgStat_Shared_Reset_Target -{ - RESET_ARCHIVER, - RESET_BGWRITER, - RESET_WAL -} PgStat_Shared_Reset_Target; - -/* Possible object types for resetting single counters */ -typedef enum PgStat_Single_Reset_Type -{ - RESET_TABLE, - RESET_FUNCTION -} PgStat_Single_Reset_Type; - /* ------------------------------------------------------------ * Structures kept in backend local memory while accumulating counts @@ -418,7 +403,7 @@ typedef struct PgStat_MsgResetcounter typedef struct PgStat_MsgResetsharedcounter { PgStat_MsgHdr m_hdr; - PgStat_Shared_Reset_Target m_resettarget; + PgStat_Kind m_resettarget; } PgStat_MsgResetsharedcounter; /* ---------- @@ -430,7 +415,7 @@ typedef struct PgStat_MsgResetsinglecounter { PgStat_MsgHdr m_hdr; Oid m_databaseid; - PgStat_Single_Reset_Type m_resettype; + PgStat_Kind m_resettype; Oid m_objectid; } PgStat_MsgResetsinglecounter; @@ -1009,8 +994,8 @@ extern void pgstat_vacuum_stat(void); extern void pgstat_ping(void); extern void pgstat_reset_counters(void); -extern void pgstat_reset_single_counter(Oid objectid, PgStat_Single_Reset_Type type); -extern void pgstat_reset_shared_counters(const char *); +extern void pgstat_reset_single_counter(PgStat_Kind kind, Oid objectid); +extern void pgstat_reset_shared_counters(PgStat_Kind kind); /* stats accessors */ extern PgStat_ArchiverStats *pgstat_fetch_stat_archiver(void); @@ -1155,6 +1140,7 @@ extern PgStat_TableStatus *find_tabstat_entry(Oid rel_id); * Functions in pgstat_replslot.c */ +extern void pgstat_reset_replslot_counters(void); extern void pgstat_reset_replslot_counter(const char *name); extern void pgstat_report_replslot(const PgStat_StatReplSlotEntry *repSlotStat); extern void pgstat_report_replslot_create(const char *slotname); @@ -1181,6 +1167,7 @@ extern int pgstat_slru_index(const char *name); * Functions in pgstat_subscription.c */ +extern void pgstat_reset_subscription_counters(void); extern void pgstat_reset_subscription_counter(Oid subid); extern void pgstat_report_subscription_error(Oid subid, bool is_apply_error); extern void pgstat_report_subscription_drop(Oid subid); diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index 8cbed1d1bc3..c9714b9f68f 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/postmaster/pgstat.c @@ -1090,7 +1090,7 @@ pgstat_reset_counters(void) * GRANT system. */ void -pgstat_reset_single_counter(Oid objoid, PgStat_Single_Reset_Type type) +pgstat_reset_single_counter(PgStat_Kind kind, Oid objoid) { PgStat_MsgResetsinglecounter msg; @@ -1099,7 +1099,7 @@ pgstat_reset_single_counter(Oid objoid, PgStat_Single_Reset_Type type) pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSINGLECOUNTER); msg.m_databaseid = MyDatabaseId; - msg.m_resettype = type; + msg.m_resettype = kind; msg.m_objectid = objoid; pgstat_send(&msg, sizeof(msg)); @@ -1112,26 +1112,15 @@ pgstat_reset_single_counter(Oid objoid, PgStat_Single_Reset_Type type) * GRANT system. */ void -pgstat_reset_shared_counters(const char *target) +pgstat_reset_shared_counters(PgStat_Kind kind) { PgStat_MsgResetsharedcounter msg; if (pgStatSock == PGINVALID_SOCKET) return; - if (strcmp(target, "archiver") == 0) - msg.m_resettarget = RESET_ARCHIVER; - else if (strcmp(target, "bgwriter") == 0) - msg.m_resettarget = RESET_BGWRITER; - else if (strcmp(target, "wal") == 0) - msg.m_resettarget = RESET_WAL; - else - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("unrecognized reset target: \"%s\"", target), - errhint("Target must be \"archiver\", \"bgwriter\", or \"wal\"."))); - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSHAREDCOUNTER); + msg.m_resettarget = kind; pgstat_send(&msg, sizeof(msg)); } @@ -3528,7 +3517,8 @@ pgstat_recv_resetcounter(PgStat_MsgResetcounter *msg, int len) static void pgstat_recv_resetsharedcounter(PgStat_MsgResetsharedcounter *msg, int len) { - if (msg->m_resettarget == RESET_BGWRITER) + if (msg->m_resettarget == PGSTAT_KIND_BGWRITER || + msg->m_resettarget == PGSTAT_KIND_CHECKPOINTER) { /* * Reset the global, bgwriter and checkpointer statistics for the @@ -3537,13 +3527,24 @@ pgstat_recv_resetsharedcounter(PgStat_MsgResetsharedcounter *msg, int len) memset(&globalStats, 0, sizeof(globalStats)); globalStats.bgwriter.stat_reset_timestamp = GetCurrentTimestamp(); } - else if (msg->m_resettarget == RESET_ARCHIVER) + else if (msg->m_resettarget == PGSTAT_KIND_ARCHIVER) { /* Reset the archiver statistics for the cluster. */ memset(&archiverStats, 0, sizeof(archiverStats)); archiverStats.stat_reset_timestamp = GetCurrentTimestamp(); } - else if (msg->m_resettarget == RESET_WAL) + else if (msg->m_resettarget == PGSTAT_KIND_SLRU) + { + int i; + TimestampTz ts = GetCurrentTimestamp(); + + for (i = 0; i < SLRU_NUM_ELEMENTS; i++) + { + memset(&slruStats[i], 0, sizeof(slruStats[i])); + slruStats[i].stat_reset_timestamp = ts; + } + } + else if (msg->m_resettarget == PGSTAT_KIND_WAL) { /* Reset the WAL statistics for the cluster. */ memset(&walStats, 0, sizeof(walStats)); @@ -3577,10 +3578,10 @@ pgstat_recv_resetsinglecounter(PgStat_MsgResetsinglecounter *msg, int len) dbentry->stat_reset_timestamp = GetCurrentTimestamp(); /* Remove object if it exists, ignore it if not */ - if (msg->m_resettype == RESET_TABLE) + if (msg->m_resettype == PGSTAT_KIND_RELATION) (void) hash_search(dbentry->tables, (void *) &(msg->m_objectid), HASH_REMOVE, NULL); - else if (msg->m_resettype == RESET_FUNCTION) + else if (msg->m_resettype == PGSTAT_KIND_FUNCTION) (void) hash_search(dbentry->functions, (void *) &(msg->m_objectid), HASH_REMOVE, NULL); } @@ -3596,8 +3597,8 @@ pgstat_recv_resetslrucounter(PgStat_MsgResetslrucounter *msg, int len) for (i = 0; i < SLRU_NUM_ELEMENTS; i++) { - /* reset entry with the given index, or all entries (index is -1) */ - if ((msg->m_index == -1) || (msg->m_index == i)) + /* reset entry with the given index */ + if (msg->m_index == i) { memset(&slruStats[i], 0, sizeof(slruStats[i])); slruStats[i].stat_reset_timestamp = ts; diff --git a/src/backend/utils/activity/pgstat_replslot.c b/src/backend/utils/activity/pgstat_replslot.c index 35078ad73c8..1c197f79a90 100644 --- a/src/backend/utils/activity/pgstat_replslot.c +++ b/src/backend/utils/activity/pgstat_replslot.c @@ -23,8 +23,23 @@ /* - * Reset counters for a single replication slot, or all replication slots - * (when name is null). + * Reset counters for all replication slots. + */ +void +pgstat_reset_replslot_counters(void) +{ + PgStat_MsgResetreplslotcounter msg; + + if (pgStatSock == PGINVALID_SOCKET) + return; + + pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETREPLSLOTCOUNTER); + msg.clearall = true; + pgstat_send(&msg, sizeof(msg)); +} + +/* + * Reset counters for a single replication slot. * * Permission checking for this function is managed through the normal * GRANT system. @@ -32,21 +47,37 @@ void pgstat_reset_replslot_counter(const char *name) { + ReplicationSlot *slot; PgStat_MsgResetreplslotcounter msg; + AssertArg(name != NULL); + if (pgStatSock == PGINVALID_SOCKET) return; - if (name) - { - namestrcpy(&msg.m_slotname, name); - msg.clearall = false; - } - else - msg.clearall = true; + /* + * Check if the slot exists with the given name. It is possible that by + * the time this message is executed the slot is dropped but at least this + * check will ensure that the given name is for a valid slot. + */ + slot = SearchNamedReplicationSlot(name, true); + + if (!slot) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("replication slot \"%s\" does not exist", + name))); + + /* + * Nothing to do for physical slots as we collect stats only for logical + * slots. + */ + if (SlotIsPhysical(slot)) + return; pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETREPLSLOTCOUNTER); - + namestrcpy(&msg.m_slotname, name); + msg.clearall = false; pgstat_send(&msg, sizeof(msg)); } diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 6dae3a5dc66..9cb9aeb890d 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -33,7 +33,7 @@ static PgStat_MsgSLRU SLRUStats[SLRU_NUM_ELEMENTS]; /* - * Reset counters for a single SLRU, or all SLRUs (when name is null). + * Reset counters for a single SLRU. * * Permission checking for this function is managed through the normal * GRANT system. @@ -43,11 +43,13 @@ pgstat_reset_slru_counter(const char *name) { PgStat_MsgResetslrucounter msg; + AssertArg(name != NULL); + if (pgStatSock == PGINVALID_SOCKET) return; pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSLRUCOUNTER); - msg.m_index = (name) ? pgstat_slru_index(name) : -1; + msg.m_index = pgstat_slru_index(name); pgstat_send(&msg, sizeof(msg)); } diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 2ee23d5ae2c..7e82bb16a19 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -21,8 +21,23 @@ /* - * Reset counters for a single subscription, or all subscriptions (when subid - * is InvalidOid). + * Reset counters for all subscriptions. + */ +void +pgstat_reset_subscription_counters(void) +{ + PgStat_MsgResetsubcounter msg; + + if (pgStatSock == PGINVALID_SOCKET) + return; + + msg.m_subid = InvalidOid; + pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSUBCOUNTER); + pgstat_send(&msg, sizeof(msg)); +} + +/* + * Reset counters for a single subscription. * * Permission checking for this function is managed through the normal * GRANT system. @@ -37,7 +52,6 @@ pgstat_reset_subscription_counter(Oid subid) msg.m_subid = subid; pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSUBCOUNTER); - pgstat_send(&msg, sizeof(msg)); } diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index fd4276fbc67..02676bf01dc 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -2075,7 +2075,24 @@ pg_stat_reset_shared(PG_FUNCTION_ARGS) { char *target = text_to_cstring(PG_GETARG_TEXT_PP(0)); - pgstat_reset_shared_counters(target); + if (strcmp(target, "archiver") == 0) + pgstat_reset_shared_counters(PGSTAT_KIND_ARCHIVER); + else if (strcmp(target, "bgwriter") == 0) + { + /* + * Historically checkpointer was part of bgwriter, continue to reset + * both for now. + */ + pgstat_reset_shared_counters(PGSTAT_KIND_BGWRITER); + pgstat_reset_shared_counters(PGSTAT_KIND_CHECKPOINTER); + } + else if (strcmp(target, "wal") == 0) + pgstat_reset_shared_counters(PGSTAT_KIND_WAL); + else + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("unrecognized reset target: \"%s\"", target), + errhint("Target must be \"archiver\", \"bgwriter\", or \"wal\"."))); PG_RETURN_VOID(); } @@ -2086,7 +2103,7 @@ pg_stat_reset_single_table_counters(PG_FUNCTION_ARGS) { Oid taboid = PG_GETARG_OID(0); - pgstat_reset_single_counter(taboid, RESET_TABLE); + pgstat_reset_single_counter(PGSTAT_KIND_RELATION, taboid); PG_RETURN_VOID(); } @@ -2096,7 +2113,7 @@ pg_stat_reset_single_function_counters(PG_FUNCTION_ARGS) { Oid funcoid = PG_GETARG_OID(0); - pgstat_reset_single_counter(funcoid, RESET_FUNCTION); + pgstat_reset_single_counter(PGSTAT_KIND_FUNCTION, funcoid); PG_RETURN_VOID(); } @@ -2107,10 +2124,13 @@ pg_stat_reset_slru(PG_FUNCTION_ARGS) { char *target = NULL; - if (!PG_ARGISNULL(0)) + if (PG_ARGISNULL(0)) + pgstat_reset_shared_counters(PGSTAT_KIND_SLRU); + else + { target = text_to_cstring(PG_GETARG_TEXT_PP(0)); - - pgstat_reset_slru_counter(target); + pgstat_reset_slru_counter(target); + } PG_RETURN_VOID(); } @@ -2121,36 +2141,14 @@ pg_stat_reset_replication_slot(PG_FUNCTION_ARGS) { char *target = NULL; - if (!PG_ARGISNULL(0)) + if (PG_ARGISNULL(0)) + pgstat_reset_replslot_counters(); + else { - ReplicationSlot *slot; - target = text_to_cstring(PG_GETARG_TEXT_PP(0)); - - /* - * Check if the slot exists with the given name. It is possible that - * by the time this message is executed the slot is dropped but at - * least this check will ensure that the given name is for a valid - * slot. - */ - slot = SearchNamedReplicationSlot(target, true); - - if (!slot) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("replication slot \"%s\" does not exist", - target))); - - /* - * Nothing to do for physical slots as we collect stats only for - * logical slots. - */ - if (SlotIsPhysical(slot)) - PG_RETURN_VOID(); + pgstat_reset_replslot_counter(target); } - pgstat_reset_replslot_counter(target); - PG_RETURN_VOID(); } @@ -2163,7 +2161,7 @@ pg_stat_reset_subscription_stats(PG_FUNCTION_ARGS) if (PG_ARGISNULL(0)) { /* Clear all subscription stats */ - subid = InvalidOid; + pgstat_reset_subscription_counters(); } else { @@ -2173,10 +2171,9 @@ pg_stat_reset_subscription_stats(PG_FUNCTION_ARGS) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("invalid subscription OID %u", subid))); + pgstat_reset_subscription_counter(subid); } - pgstat_reset_subscription_counter(subid); - PG_RETURN_VOID(); } diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index a5c15a1d68f..53b73087eb8 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1966,8 +1966,6 @@ PgStat_MsgTempFile PgStat_MsgVacuum PgStat_MsgWal PgStat_SLRUStats -PgStat_Shared_Reset_Target -PgStat_Single_Reset_Type PgStat_StatDBEntry PgStat_StatFuncEntry PgStat_StatReplSlotEntry -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0010-pgstat-scaffolding-for-transactional-stats-creat.patch" From f792dd09d9b9abf1301fb73daa14cdf87edec36c Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sun, 3 Apr 2022 12:54:14 -0700 Subject: [PATCH v68 10/31] pgstat: scaffolding for transactional stats creation / drop. On its own this doesn't achieve anything, but it's a largely independent piece of infrastructure, so committing it separately makes sense. FIXME: Need to bump XLOG_PAGE_MAGIC Author: Andres Freund Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- src/include/access/xact.h | 40 +++- src/include/pgstat.h | 15 +- src/include/utils/pgstat_internal.h | 13 ++ src/backend/access/rmgrdesc/xactdesc.c | 52 +++++ src/backend/access/transam/twophase.c | 45 ++++ src/backend/access/transam/xact.c | 64 ++++- src/backend/catalog/heap.c | 7 + src/backend/catalog/pg_proc.c | 5 + src/backend/commands/functioncmds.c | 2 + src/backend/commands/subscriptioncmds.c | 4 +- src/backend/postmaster/pgstat.c | 4 +- src/backend/replication/slot.c | 4 +- src/backend/storage/smgr/smgr.c | 5 - src/backend/utils/activity/pgstat_function.c | 22 ++ src/backend/utils/activity/pgstat_relation.c | 39 ++-- src/backend/utils/activity/pgstat_replslot.c | 4 +- .../utils/activity/pgstat_subscription.c | 21 +- src/backend/utils/activity/pgstat_xact.c | 220 ++++++++++++++++++ src/tools/pgindent/typedefs.list | 3 + 19 files changed, 525 insertions(+), 44 deletions(-) diff --git a/src/include/access/xact.h b/src/include/access/xact.h index 062cc7e17d8..4e1e8735010 100644 --- a/src/include/access/xact.h +++ b/src/include/access/xact.h @@ -180,6 +180,7 @@ typedef struct SavedTransactionCharacteristics #define XACT_XINFO_HAS_ORIGIN (1U << 5) #define XACT_XINFO_HAS_AE_LOCKS (1U << 6) #define XACT_XINFO_HAS_GID (1U << 7) +#define XACT_XINFO_HAS_DROPPED_STATS (1U << 8) /* * Also stored in xinfo, these indicating a variety of additional actions that @@ -230,7 +231,7 @@ typedef struct xl_xact_assignment typedef struct xl_xact_xinfo { /* - * Even though we right now only require 1 byte of space in xinfo we use + * Even though we right now only require two bytes of space in xinfo we use * four so following records don't have to care about alignment. Commit * records can be large, so copying large portions isn't attractive. */ @@ -257,6 +258,27 @@ typedef struct xl_xact_relfilenodes } xl_xact_relfilenodes; #define MinSizeOfXactRelfilenodes offsetof(xl_xact_relfilenodes, xnodes) +/* + * A transactionally dropped statistics entry. + * + * Declared here rather than pgstat.h because pgstat.h can't be included from + * frontend code, but the WAL format needs to be readable by frontend + * programs. + */ +typedef struct xl_xact_stats_item +{ + int kind; + Oid dboid; + Oid objoid; +} xl_xact_stats_item; + +typedef struct xl_xact_stats_items +{ + int nitems; + xl_xact_stats_item items[FLEXIBLE_ARRAY_MEMBER]; +} xl_xact_stats_items; +#define MinSizeOfXactStatsItems offsetof(xl_xact_stats_items, items) + typedef struct xl_xact_invals { int nmsgs; /* number of shared inval msgs */ @@ -283,6 +305,7 @@ typedef struct xl_xact_commit /* xl_xact_dbinfo follows if XINFO_HAS_DBINFO */ /* xl_xact_subxacts follows if XINFO_HAS_SUBXACT */ /* xl_xact_relfilenodes follows if XINFO_HAS_RELFILENODES */ + /* xl_xact_stats_items follows if XINFO_HAS_DROPPED_STATS */ /* xl_xact_invals follows if XINFO_HAS_INVALS */ /* xl_xact_twophase follows if XINFO_HAS_TWOPHASE */ /* twophase_gid follows if XINFO_HAS_GID. As a null-terminated string. */ @@ -298,6 +321,7 @@ typedef struct xl_xact_abort /* xl_xact_dbinfo follows if XINFO_HAS_DBINFO */ /* xl_xact_subxacts follows if XINFO_HAS_SUBXACT */ /* xl_xact_relfilenodes follows if XINFO_HAS_RELFILENODES */ + /* xl_xact_stats_items follows if XINFO_HAS_DROPPED_STATS */ /* No invalidation messages needed. */ /* xl_xact_twophase follows if XINFO_HAS_TWOPHASE */ /* twophase_gid follows if XINFO_HAS_GID. As a null-terminated string. */ @@ -316,6 +340,8 @@ typedef struct xl_xact_prepare int32 nsubxacts; /* number of following subxact XIDs */ int32 ncommitrels; /* number of delete-on-commit rels */ int32 nabortrels; /* number of delete-on-abort rels */ + int32 ncommitstats; /* number of stats to drop on commit */ + int32 nabortstats; /* number of stats to drop on abort */ int32 ninvalmsgs; /* number of cache invalidation messages */ bool initfileinval; /* does relcache init file need invalidation? */ uint16 gidlen; /* length of the GID - GID follows the header */ @@ -342,6 +368,9 @@ typedef struct xl_xact_parsed_commit int nrels; RelFileNode *xnodes; + int nstats; + xl_xact_stats_item *stats; + int nmsgs; SharedInvalidationMessage *msgs; @@ -349,6 +378,8 @@ typedef struct xl_xact_parsed_commit char twophase_gid[GIDSIZE]; /* only for 2PC */ int nabortrels; /* only for 2PC */ RelFileNode *abortnodes; /* only for 2PC */ + int nabortstats; /* only for 2PC */ + xl_xact_stats_item *abortstats; /* only for 2PC */ XLogRecPtr origin_lsn; TimestampTz origin_timestamp; @@ -370,6 +401,9 @@ typedef struct xl_xact_parsed_abort int nrels; RelFileNode *xnodes; + int nstats; + xl_xact_stats_item *stats; + TransactionId twophase_xid; /* only for 2PC */ char twophase_gid[GIDSIZE]; /* only for 2PC */ @@ -449,6 +483,8 @@ extern int xactGetCommittedChildren(TransactionId **ptr); extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time, int nsubxacts, TransactionId *subxacts, int nrels, RelFileNode *rels, + int nstats, + xl_xact_stats_item *stats, int nmsgs, SharedInvalidationMessage *msgs, bool relcacheInval, int xactflags, @@ -458,6 +494,8 @@ extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time, extern XLogRecPtr XactLogAbortRecord(TimestampTz abort_time, int nsubxacts, TransactionId *subxacts, int nrels, RelFileNode *rels, + int nstats, + xl_xact_stats_item *stats, int xactflags, TransactionId twophase_xid, const char *twophase_gid); extern void xact_redo(XLogReaderState *record); diff --git a/src/include/pgstat.h b/src/include/pgstat.h index a87b45f368e..4e0692d8b2b 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -987,6 +987,9 @@ extern void AtEOSubXact_PgStat(bool isCommit, int nestDepth); extern void AtPrepare_PgStat(void); extern void PostPrepare_PgStat(void); extern void pgstat_clear_snapshot(void); +struct xl_xact_stats_item; +extern int pgstat_get_transactional_drops(bool isCommit, struct xl_xact_stats_item **items); +extern void pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_item *items, bool is_redo); /* Functions called from backends */ extern void pgstat_report_stat(bool force); @@ -1057,6 +1060,9 @@ extern void pgstat_report_connect(Oid dboid); * Functions in pgstat_function.c */ +extern void pgstat_create_function(Oid proid); +extern void pgstat_drop_function(Oid proid); + struct FunctionCallInfoBaseData; extern void pgstat_init_function_usage(struct FunctionCallInfoBaseData *fcinfo, PgStat_FunctionCallUsage *fcu); @@ -1070,6 +1076,8 @@ extern PgStat_BackendFunctionEntry *find_funcstat_entry(Oid func_id); * Functions in pgstat_relation.c */ +extern void pgstat_create_relation(Relation rel); +extern void pgstat_drop_relation(Relation rel); extern void pgstat_copy_relation_stats(Relation dstrel, Relation srcrel); extern void pgstat_relation_init(Relation rel); @@ -1143,8 +1151,8 @@ extern PgStat_TableStatus *find_tabstat_entry(Oid rel_id); extern void pgstat_reset_replslot_counters(void); extern void pgstat_reset_replslot_counter(const char *name); extern void pgstat_report_replslot(const PgStat_StatReplSlotEntry *repSlotStat); -extern void pgstat_report_replslot_create(const char *slotname); -extern void pgstat_report_replslot_drop(const char *slotname); +extern void pgstat_create_replslot(const char *slotname); +extern void pgstat_drop_replslot(const char *slotname); /* @@ -1170,7 +1178,8 @@ extern int pgstat_slru_index(const char *name); extern void pgstat_reset_subscription_counters(void); extern void pgstat_reset_subscription_counter(Oid subid); extern void pgstat_report_subscription_error(Oid subid, bool is_apply_error); -extern void pgstat_report_subscription_drop(Oid subid); +extern void pgstat_create_subscription(Oid subid); +extern void pgstat_drop_subscription(Oid subid); /* diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 7f4d79470f7..16e99afa885 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -42,6 +42,16 @@ typedef struct PgStat_SubXactStatus struct PgStat_SubXactStatus *prev; /* higher-level subxact if any */ + /* + * Dropping the statistics for objects that dropped transactionally itself + * needs to be transactional. Therefore we collect the stats dropped in + * the current (sub-)transaction and only execute the stats drop when we + * know if the transaction commits/aborts. To handle replicas and crashes, + * stats drops are included in commit records. + */ + dlist_head pending_drops; + int pending_drops_count; + /* * Tuple insertion/deletion counts for an open transaction can't be * propagated into PgStat_TableStatus counters until we know if it is @@ -134,6 +144,9 @@ extern bool pgstat_wal_pending(void); */ extern PgStat_SubXactStatus *pgstat_xact_stack_level_get(int nest_level); +extern void pgstat_drop_transactional(PgStat_Kind kind, Oid dboid, Oid objoid); +extern void pgstat_create_transactional(PgStat_Kind kind, Oid dboid, Oid objoid); + /* * Variables in pgstat.c diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c index 025d556f6ce..d3f625d0726 100644 --- a/src/backend/access/rmgrdesc/xactdesc.c +++ b/src/backend/access/rmgrdesc/xactdesc.c @@ -84,6 +84,17 @@ ParseCommitRecord(uint8 info, xl_xact_commit *xlrec, xl_xact_parsed_commit *pars data += xl_relfilenodes->nrels * sizeof(RelFileNode); } + if (parsed->xinfo & XACT_XINFO_HAS_DROPPED_STATS) + { + xl_xact_stats_items *xl_drops = (xl_xact_stats_items *) data; + + parsed->nstats = xl_drops->nitems; + parsed->stats = xl_drops->items; + + data += MinSizeOfXactStatsItems; + data += xl_drops->nitems * sizeof(xl_xact_stats_item); + } + if (parsed->xinfo & XACT_XINFO_HAS_INVALS) { xl_xact_invals *xl_invals = (xl_xact_invals *) data; @@ -179,6 +190,17 @@ ParseAbortRecord(uint8 info, xl_xact_abort *xlrec, xl_xact_parsed_abort *parsed) data += xl_relfilenodes->nrels * sizeof(RelFileNode); } + if (parsed->xinfo & XACT_XINFO_HAS_DROPPED_STATS) + { + xl_xact_stats_items *xl_drops = (xl_xact_stats_items *) data; + + parsed->nstats = xl_drops->nitems; + parsed->stats = xl_drops->items; + + data += MinSizeOfXactStatsItems; + data += xl_drops->nitems * sizeof(xl_xact_stats_item); + } + if (parsed->xinfo & XACT_XINFO_HAS_TWOPHASE) { xl_xact_twophase *xl_twophase = (xl_xact_twophase *) data; @@ -244,6 +266,12 @@ ParsePrepareRecord(uint8 info, xl_xact_prepare *xlrec, xl_xact_parsed_prepare *p parsed->abortnodes = (RelFileNode *) bufptr; bufptr += MAXALIGN(xlrec->nabortrels * sizeof(RelFileNode)); + parsed->stats = (xl_xact_stats_item *) bufptr; + bufptr += MAXALIGN(xlrec->ncommitstats * sizeof(xl_xact_stats_item)); + + parsed->abortstats = (xl_xact_stats_item *) bufptr; + bufptr += MAXALIGN(xlrec->nabortstats * sizeof(xl_xact_stats_item)); + parsed->msgs = (SharedInvalidationMessage *) bufptr; bufptr += MAXALIGN(xlrec->ninvalmsgs * sizeof(SharedInvalidationMessage)); } @@ -280,6 +308,25 @@ xact_desc_subxacts(StringInfo buf, int nsubxacts, TransactionId *subxacts) } } +static void +xact_desc_stats(StringInfo buf, const char *label, + int ndropped, xl_xact_stats_item *dropped_stats) +{ + int i; + + if (ndropped > 0) + { + appendStringInfo(buf, "; %sdropped stats:", label); + for (i = 0; i < ndropped; i++) + { + appendStringInfo(buf, " %u/%u/%u", + dropped_stats[i].kind, + dropped_stats[i].dboid, + dropped_stats[i].objoid); + } + } +} + static void xact_desc_commit(StringInfo buf, uint8 info, xl_xact_commit *xlrec, RepOriginId origin_id) { @@ -295,6 +342,7 @@ xact_desc_commit(StringInfo buf, uint8 info, xl_xact_commit *xlrec, RepOriginId xact_desc_relations(buf, "rels", parsed.nrels, parsed.xnodes); xact_desc_subxacts(buf, parsed.nsubxacts, parsed.subxacts); + xact_desc_stats(buf, "", parsed.nstats, parsed.stats); standby_desc_invalidations(buf, parsed.nmsgs, parsed.msgs, parsed.dbId, parsed.tsId, @@ -338,6 +386,8 @@ xact_desc_abort(StringInfo buf, uint8 info, xl_xact_abort *xlrec, RepOriginId or LSN_FORMAT_ARGS(parsed.origin_lsn), timestamptz_to_str(parsed.origin_timestamp)); } + + xact_desc_stats(buf, "", parsed.nstats, parsed.stats); } static void @@ -353,6 +403,8 @@ xact_desc_prepare(StringInfo buf, uint8 info, xl_xact_prepare *xlrec, RepOriginI xact_desc_relations(buf, "rels(commit)", parsed.nrels, parsed.xnodes); xact_desc_relations(buf, "rels(abort)", parsed.nabortrels, parsed.abortnodes); + xact_desc_stats(buf, "commit ", parsed.nstats, parsed.stats); + xact_desc_stats(buf, "abort ", parsed.nabortstats, parsed.abortstats); xact_desc_subxacts(buf, parsed.nsubxacts, parsed.subxacts); standby_desc_invalidations(buf, parsed.nmsgs, parsed.msgs, parsed.dbId, diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c index 4dc8ccc12b9..b35da6f1aad 100644 --- a/src/backend/access/transam/twophase.c +++ b/src/backend/access/transam/twophase.c @@ -205,6 +205,8 @@ static void RecordTransactionCommitPrepared(TransactionId xid, TransactionId *children, int nrels, RelFileNode *rels, + int nstats, + xl_xact_stats_item *stats, int ninvalmsgs, SharedInvalidationMessage *invalmsgs, bool initfileinval, @@ -214,6 +216,8 @@ static void RecordTransactionAbortPrepared(TransactionId xid, TransactionId *children, int nrels, RelFileNode *rels, + int nstats, + xl_xact_stats_item *stats, const char *gid); static void ProcessRecords(char *bufptr, TransactionId xid, const TwoPhaseCallback callbacks[]); @@ -1046,6 +1050,8 @@ StartPrepare(GlobalTransaction gxact) TransactionId *children; RelFileNode *commitrels; RelFileNode *abortrels; + xl_xact_stats_item *abortstats = NULL; + xl_xact_stats_item *commitstats = NULL; SharedInvalidationMessage *invalmsgs; /* Initialize linked list */ @@ -1071,6 +1077,10 @@ StartPrepare(GlobalTransaction gxact) hdr.nsubxacts = xactGetCommittedChildren(&children); hdr.ncommitrels = smgrGetPendingDeletes(true, &commitrels); hdr.nabortrels = smgrGetPendingDeletes(false, &abortrels); + hdr.ncommitstats = + pgstat_get_transactional_drops(true, &commitstats); + hdr.nabortstats = + pgstat_get_transactional_drops(false, &abortstats); hdr.ninvalmsgs = xactGetCommittedInvalidationMessages(&invalmsgs, &hdr.initfileinval); hdr.gidlen = strlen(gxact->gid) + 1; /* Include '\0' */ @@ -1101,6 +1111,18 @@ StartPrepare(GlobalTransaction gxact) save_state_data(abortrels, hdr.nabortrels * sizeof(RelFileNode)); pfree(abortrels); } + if (hdr.ncommitstats > 0) + { + save_state_data(commitstats, + hdr.ncommitstats * sizeof(xl_xact_stats_item)); + pfree(commitstats); + } + if (hdr.nabortstats > 0) + { + save_state_data(abortstats, + hdr.nabortstats * sizeof(xl_xact_stats_item)); + pfree(abortstats); + } if (hdr.ninvalmsgs > 0) { save_state_data(invalmsgs, @@ -1472,6 +1494,8 @@ FinishPreparedTransaction(const char *gid, bool isCommit) RelFileNode *abortrels; RelFileNode *delrels; int ndelrels; + xl_xact_stats_item *commitstats; + xl_xact_stats_item *abortstats; SharedInvalidationMessage *invalmsgs; /* @@ -1506,6 +1530,10 @@ FinishPreparedTransaction(const char *gid, bool isCommit) bufptr += MAXALIGN(hdr->ncommitrels * sizeof(RelFileNode)); abortrels = (RelFileNode *) bufptr; bufptr += MAXALIGN(hdr->nabortrels * sizeof(RelFileNode)); + commitstats = (xl_xact_stats_item*) bufptr; + bufptr += MAXALIGN(hdr->ncommitstats * sizeof(xl_xact_stats_item)); + abortstats = (xl_xact_stats_item*) bufptr; + bufptr += MAXALIGN(hdr->nabortstats * sizeof(xl_xact_stats_item)); invalmsgs = (SharedInvalidationMessage *) bufptr; bufptr += MAXALIGN(hdr->ninvalmsgs * sizeof(SharedInvalidationMessage)); @@ -1527,12 +1555,16 @@ FinishPreparedTransaction(const char *gid, bool isCommit) RecordTransactionCommitPrepared(xid, hdr->nsubxacts, children, hdr->ncommitrels, commitrels, + hdr->ncommitstats, + commitstats, hdr->ninvalmsgs, invalmsgs, hdr->initfileinval, gid); else RecordTransactionAbortPrepared(xid, hdr->nsubxacts, children, hdr->nabortrels, abortrels, + hdr->nabortstats, + abortstats, gid); ProcArrayRemove(proc, latestXid); @@ -1568,6 +1600,11 @@ FinishPreparedTransaction(const char *gid, bool isCommit) /* Make sure files supposed to be dropped are dropped */ DropRelationFiles(delrels, ndelrels, false); + if (isCommit) + pgstat_execute_transactional_drops(hdr->ncommitstats, commitstats, false); + else + pgstat_execute_transactional_drops(hdr->nabortstats, abortstats, false); + /* * Handle cache invalidation messages. * @@ -2066,6 +2103,8 @@ RecoverPreparedTransactions(void) bufptr += MAXALIGN(hdr->nsubxacts * sizeof(TransactionId)); bufptr += MAXALIGN(hdr->ncommitrels * sizeof(RelFileNode)); bufptr += MAXALIGN(hdr->nabortrels * sizeof(RelFileNode)); + bufptr += MAXALIGN(hdr->ncommitstats * sizeof(xl_xact_stats_item)); + bufptr += MAXALIGN(hdr->nabortstats * sizeof(xl_xact_stats_item)); bufptr += MAXALIGN(hdr->ninvalmsgs * sizeof(SharedInvalidationMessage)); /* @@ -2248,6 +2287,8 @@ RecordTransactionCommitPrepared(TransactionId xid, TransactionId *children, int nrels, RelFileNode *rels, + int nstats, + xl_xact_stats_item *stats, int ninvalmsgs, SharedInvalidationMessage *invalmsgs, bool initfileinval, @@ -2277,6 +2318,7 @@ RecordTransactionCommitPrepared(TransactionId xid, */ recptr = XactLogCommitRecord(committs, nchildren, children, nrels, rels, + nstats, stats, ninvalmsgs, invalmsgs, initfileinval, MyXactFlags | XACT_FLAGS_ACQUIREDACCESSEXCLUSIVELOCK, @@ -2343,6 +2385,8 @@ RecordTransactionAbortPrepared(TransactionId xid, TransactionId *children, int nrels, RelFileNode *rels, + int nstats, + xl_xact_stats_item *stats, const char *gid) { XLogRecPtr recptr; @@ -2373,6 +2417,7 @@ RecordTransactionAbortPrepared(TransactionId xid, recptr = XactLogAbortRecord(GetCurrentTimestamp(), nchildren, children, nrels, rels, + nstats, stats, MyXactFlags | XACT_FLAGS_ACQUIREDACCESSEXCLUSIVELOCK, xid, gid); diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 3596a7d7345..c3e7f605765 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -1285,6 +1285,8 @@ RecordTransactionCommit(void) RelFileNode *rels; int nchildren; TransactionId *children; + int ndroppedstats = 0; + xl_xact_stats_item *droppedstats = NULL; int nmsgs = 0; SharedInvalidationMessage *invalMessages = NULL; bool RelcacheInitFileInval = false; @@ -1303,6 +1305,7 @@ RecordTransactionCommit(void) /* Get data needed for commit record */ nrels = smgrGetPendingDeletes(true, &rels); nchildren = xactGetCommittedChildren(&children); + ndroppedstats = pgstat_get_transactional_drops(true, &droppedstats); if (XLogStandbyInfoActive()) nmsgs = xactGetCommittedInvalidationMessages(&invalMessages, &RelcacheInitFileInval); @@ -1317,10 +1320,12 @@ RecordTransactionCommit(void) /* * We expect that every RelationDropStorage is followed by a catalog * update, and hence XID assignment, so we shouldn't get here with any - * pending deletes. Use a real test not just an Assert to check this, - * since it's a bit fragile. + * pending deletes. Same is true for dropping stats. + * + * Use a real test not just an Assert to check this, since it's a bit + * fragile. */ - if (nrels != 0) + if (nrels != 0 || ndroppedstats != 0) elog(ERROR, "cannot commit a transaction that deleted files but has no xid"); /* Can't have child XIDs either; AssignTransactionId enforces this */ @@ -1395,6 +1400,7 @@ RecordTransactionCommit(void) XactLogCommitRecord(xactStopTimestamp, nchildren, children, nrels, rels, + ndroppedstats, droppedstats, nmsgs, invalMessages, RelcacheInitFileInval, MyXactFlags, @@ -1698,6 +1704,8 @@ RecordTransactionAbort(bool isSubXact) TransactionId latestXid; int nrels; RelFileNode *rels; + int ndroppedstats = 0; + xl_xact_stats_item *droppedstats = NULL; int nchildren; TransactionId *children; TimestampTz xact_time; @@ -1734,6 +1742,7 @@ RecordTransactionAbort(bool isSubXact) /* Fetch the data we need for the abort record */ nrels = smgrGetPendingDeletes(false, &rels); nchildren = xactGetCommittedChildren(&children); + ndroppedstats = pgstat_get_transactional_drops(false, &droppedstats); /* XXX do we really need a critical section here? */ START_CRIT_SECTION(); @@ -1750,6 +1759,7 @@ RecordTransactionAbort(bool isSubXact) XactLogAbortRecord(xact_time, nchildren, children, nrels, rels, + ndroppedstats, droppedstats, MyXactFlags, InvalidTransactionId, NULL); @@ -5573,6 +5583,7 @@ XLogRecPtr XactLogCommitRecord(TimestampTz commit_time, int nsubxacts, TransactionId *subxacts, int nrels, RelFileNode *rels, + int ndroppedstats, xl_xact_stats_item *droppedstats, int nmsgs, SharedInvalidationMessage *msgs, bool relcacheInval, int xactflags, TransactionId twophase_xid, @@ -5583,6 +5594,7 @@ XactLogCommitRecord(TimestampTz commit_time, xl_xact_dbinfo xl_dbinfo; xl_xact_subxacts xl_subxacts; xl_xact_relfilenodes xl_relfilenodes; + xl_xact_stats_items xl_dropped_stats; xl_xact_invals xl_invals; xl_xact_twophase xl_twophase; xl_xact_origin xl_origin; @@ -5640,6 +5652,12 @@ XactLogCommitRecord(TimestampTz commit_time, info |= XLR_SPECIAL_REL_UPDATE; } + if (ndroppedstats > 0) + { + xl_xinfo.xinfo |= XACT_XINFO_HAS_DROPPED_STATS; + xl_dropped_stats.nitems = ndroppedstats; + } + if (nmsgs > 0) { xl_xinfo.xinfo |= XACT_XINFO_HAS_INVALS; @@ -5696,6 +5714,14 @@ XactLogCommitRecord(TimestampTz commit_time, nrels * sizeof(RelFileNode)); } + if (xl_xinfo.xinfo & XACT_XINFO_HAS_DROPPED_STATS) + { + XLogRegisterData((char *) (&xl_dropped_stats), + MinSizeOfXactStatsItems); + XLogRegisterData((char *) droppedstats, + ndroppedstats * sizeof(xl_xact_stats_item)); + } + if (xl_xinfo.xinfo & XACT_XINFO_HAS_INVALS) { XLogRegisterData((char *) (&xl_invals), MinSizeOfXactInvals); @@ -5729,6 +5755,7 @@ XLogRecPtr XactLogAbortRecord(TimestampTz abort_time, int nsubxacts, TransactionId *subxacts, int nrels, RelFileNode *rels, + int ndroppedstats, xl_xact_stats_item *droppedstats, int xactflags, TransactionId twophase_xid, const char *twophase_gid) { @@ -5736,6 +5763,7 @@ XactLogAbortRecord(TimestampTz abort_time, xl_xact_xinfo xl_xinfo; xl_xact_subxacts xl_subxacts; xl_xact_relfilenodes xl_relfilenodes; + xl_xact_stats_items xl_dropped_stats; xl_xact_twophase xl_twophase; xl_xact_dbinfo xl_dbinfo; xl_xact_origin xl_origin; @@ -5773,6 +5801,12 @@ XactLogAbortRecord(TimestampTz abort_time, info |= XLR_SPECIAL_REL_UPDATE; } + if (ndroppedstats > 0) + { + xl_xinfo.xinfo |= XACT_XINFO_HAS_DROPPED_STATS; + xl_dropped_stats.nitems = ndroppedstats; + } + if (TransactionIdIsValid(twophase_xid)) { xl_xinfo.xinfo |= XACT_XINFO_HAS_TWOPHASE; @@ -5834,6 +5868,14 @@ XactLogAbortRecord(TimestampTz abort_time, nrels * sizeof(RelFileNode)); } + if (xl_xinfo.xinfo & XACT_XINFO_HAS_DROPPED_STATS) + { + XLogRegisterData((char *) (&xl_dropped_stats), + MinSizeOfXactStatsItems); + XLogRegisterData((char *) droppedstats, + ndroppedstats * sizeof(xl_xact_stats_item)); + } + if (xl_xinfo.xinfo & XACT_XINFO_HAS_TWOPHASE) { XLogRegisterData((char *) (&xl_twophase), sizeof(xl_xact_twophase)); @@ -5967,6 +6009,14 @@ xact_redo_commit(xl_xact_parsed_commit *parsed, DropRelationFiles(parsed->xnodes, parsed->nrels, true); } + if (parsed->nstats > 0) + { + /* see equivalent call for relations above */ + XLogFlush(lsn); + + pgstat_execute_transactional_drops(parsed->nstats, parsed->stats, true); + } + /* * We issue an XLogFlush() for the same reason we emit ForceSyncCommit() * in normal operation. For example, in CREATE DATABASE, we copy all files @@ -6069,6 +6119,14 @@ xact_redo_abort(xl_xact_parsed_abort *parsed, TransactionId xid, DropRelationFiles(parsed->xnodes, parsed->nrels, true); } + + if (parsed->nstats > 0) + { + /* see equivalent call for relations above */ + XLogFlush(lsn); + + pgstat_execute_transactional_drops(parsed->nstats, parsed->stats, true); + } } void diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 6eb78a9c0ff..9b512ccd3c0 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -64,6 +64,7 @@ #include "parser/parse_relation.h" #include "parser/parsetree.h" #include "partitioning/partdesc.h" +#include "pgstat.h" #include "storage/lmgr.h" #include "storage/predicate.h" #include "utils/builtins.h" @@ -1475,6 +1476,9 @@ heap_create_with_catalog(const char *relname, if (oncommit != ONCOMMIT_NOOP) register_on_commit_action(relid, oncommit); + /* ensure that stats are dropped if transaction aborts */ + pgstat_create_relation(new_rel_desc); + /* * ok, the relation has been cataloged, so close our relations and return * the OID of the newly created relation. @@ -1851,6 +1855,9 @@ heap_drop_with_catalog(Oid relid) if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind)) RelationDropStorage(rel); + /* ensure that stats are dropped if transaction commits */ + pgstat_drop_relation(rel); + /* * Close relcache entry, but *keep* AccessExclusiveLock on the relation * until transaction commit. This ensures no one else will try to do diff --git a/src/backend/catalog/pg_proc.c b/src/backend/catalog/pg_proc.c index ac8aacbd591..d82221fdb8f 100644 --- a/src/backend/catalog/pg_proc.c +++ b/src/backend/catalog/pg_proc.c @@ -35,6 +35,7 @@ #include "parser/analyze.h" #include "parser/parse_coerce.h" #include "parser/parse_type.h" +#include "pgstat.h" #include "rewrite/rewriteHandler.h" #include "tcop/pquery.h" #include "tcop/tcopprot.h" @@ -709,6 +710,10 @@ ProcedureCreate(const char *procedureName, AtEOXact_GUC(true, save_nestlevel); } + /* ensure that stats are dropped if transaction commits */ + if (!is_update) + pgstat_create_function(retval); + return myself; } diff --git a/src/backend/commands/functioncmds.c b/src/backend/commands/functioncmds.c index 25b75375a8e..91f02a7eb2c 100644 --- a/src/backend/commands/functioncmds.c +++ b/src/backend/commands/functioncmds.c @@ -1325,6 +1325,8 @@ RemoveFunctionById(Oid funcOid) table_close(relation, RowExclusiveLock); + pgstat_drop_function(funcOid); + /* * If there's a pg_aggregate tuple, delete that too. */ diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 51505373ea4..83192dbd51f 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -738,6 +738,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, table_close(rel, RowExclusiveLock); + pgstat_create_subscription(subid); + if (opts.enabled) ApplyLauncherWakeupAtCommit(); @@ -1592,7 +1594,7 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel) * gets lost. */ if (slotname) - pgstat_report_subscription_drop(subid); + pgstat_drop_subscription(subid); table_close(rel, NoLock); } diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index c9714b9f68f..2ce4ffd2b64 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/postmaster/pgstat.c @@ -862,7 +862,7 @@ pgstat_vacuum_stat(void) CHECK_FOR_INTERRUPTS(); if (SearchNamedReplicationSlot(NameStr(slotentry->slotname), true) == NULL) - pgstat_report_replslot_drop(NameStr(slotentry->slotname)); + pgstat_drop_replslot(NameStr(slotentry->slotname)); } } @@ -886,7 +886,7 @@ pgstat_vacuum_stat(void) CHECK_FOR_INTERRUPTS(); if (hash_search(htab, (void *) &(subentry->subid), HASH_FIND, NULL) == NULL) - pgstat_report_subscription_drop(subentry->subid); + pgstat_drop_subscription(subentry->subid); } hash_destroy(htab); diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index ed4c8b3ad55..b02571e4dd8 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -356,7 +356,7 @@ ReplicationSlotCreate(const char *name, bool db_specific, * ReplicationSlotAllocationLock. */ if (SlotIsLogical(slot)) - pgstat_report_replslot_create(NameStr(slot->data.name)); + pgstat_create_replslot(NameStr(slot->data.name)); /* * Now that the slot has been marked as in_use and active, it's safe to @@ -746,7 +746,7 @@ ReplicationSlotDropPtr(ReplicationSlot *slot) * doesn't seem worth doing as in practice this won't happen frequently. */ if (SlotIsLogical(slot)) - pgstat_report_replslot_drop(NameStr(slot->data.name)); + pgstat_drop_replslot(NameStr(slot->data.name)); /* * We release this at the very end, so that nobody starts trying to create diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c index d71a557a352..2c7a2b28572 100644 --- a/src/backend/storage/smgr/smgr.c +++ b/src/backend/storage/smgr/smgr.c @@ -415,11 +415,6 @@ smgrdounlinkall(SMgrRelation *rels, int nrels, bool isRedo) smgrsw[which].smgr_close(rels[i], forknum); } - /* - * It'd be nice to tell the stats collector to forget them immediately, - * too. But we can't because we don't know the OIDs. - */ - /* * Send a shared-inval message to force other backends to close any * dangling smgr references they may have for these rels. We should do diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index 93ec29757aa..ad9879afb2a 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -48,6 +48,28 @@ static HTAB *pgStatFunctions = NULL; static instr_time total_func_time; +/* + * Ensure that stats are dropped if transaction aborts. + */ +void +pgstat_create_function(Oid proid) +{ + pgstat_create_transactional(PGSTAT_KIND_FUNCTION, + MyDatabaseId, + proid); +} + +/* + * Ensure that stats are dropped if transaction commits. + */ +void +pgstat_drop_function(Oid proid) +{ + pgstat_drop_transactional(PGSTAT_KIND_FUNCTION, + MyDatabaseId, + proid); +} + /* * Initialize function call usage data. * Called by the executor before invoking a function. diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index b0ac406e5fd..53080eadb89 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -171,33 +171,26 @@ pgstat_relation_init(Relation rel) } /* - * Tell the collector that we just dropped a relation. - * (If the message gets lost, we will still clean the dead entry eventually - * via future invocations of pgstat_vacuum_stat().) - * - * Currently not used for lack of any good place to call it; we rely - * entirely on pgstat_vacuum_stat() to clean out stats for dead rels. + * Ensure that stats are dropped if transaction aborts. */ -#ifdef NOT_USED void -pgstat_drop_relation(Oid relid) +pgstat_create_relation(Relation rel) { - PgStat_MsgTabpurge msg; - int len; - - if (pgStatSock == PGINVALID_SOCKET) - return; - - msg.m_tableid[0] = relid; - msg.m_nentries = 1; - - len = offsetof(PgStat_MsgTabpurge, m_tableid[0]) + sizeof(Oid); - - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_TABPURGE); - msg.m_databaseid = MyDatabaseId; - pgstat_send(&msg, len); + pgstat_create_transactional(PGSTAT_KIND_RELATION, + rel->rd_rel->relisshared ? InvalidOid : MyDatabaseId, + RelationGetRelid(rel)); +} + +/* + * Ensure that stats are dropped if transaction commits. + */ +void +pgstat_drop_relation(Relation rel) +{ + pgstat_drop_transactional(PGSTAT_KIND_RELATION, + rel->rd_rel->relisshared ? InvalidOid : MyDatabaseId, + RelationGetRelid(rel)); } -#endif /* NOT_USED */ /* * Called from autovacuum.c to report startup of an autovacuum process. diff --git a/src/backend/utils/activity/pgstat_replslot.c b/src/backend/utils/activity/pgstat_replslot.c index 1c197f79a90..8d64ecd8aaf 100644 --- a/src/backend/utils/activity/pgstat_replslot.c +++ b/src/backend/utils/activity/pgstat_replslot.c @@ -111,7 +111,7 @@ pgstat_report_replslot(const PgStat_StatReplSlotEntry *repSlotStat) * Report replication slot creation. */ void -pgstat_report_replslot_create(const char *slotname) +pgstat_create_replslot(const char *slotname) { PgStat_MsgReplSlot msg; @@ -126,7 +126,7 @@ pgstat_report_replslot_create(const char *slotname) * Report replication slot drop. */ void -pgstat_report_replslot_drop(const char *slotname) +pgstat_drop_replslot(const char *slotname) { PgStat_MsgReplSlot msg; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 7e82bb16a19..d1d8c0ecfd2 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -70,14 +70,31 @@ pgstat_report_subscription_error(Oid subid, bool is_apply_error) } /* - * Report dropping the subscription. + * Report creating the subscription. + * + * Ensures that stats are dropped if transaction rolls back. */ void -pgstat_report_subscription_drop(Oid subid) +pgstat_create_subscription(Oid subid) +{ + pgstat_create_transactional(PGSTAT_KIND_SUBSCRIPTION, + InvalidOid, subid); +} + +/* + * Report dropping the subscription. + * + * Ensures that stats are dropped if transaction commits. + */ +void +pgstat_drop_subscription(Oid subid) { PgStat_MsgSubscriptionDrop msg; pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_SUBSCRIPTIONDROP); msg.m_subid = subid; pgstat_send(&msg, sizeof(PgStat_MsgSubscriptionDrop)); + + pgstat_drop_transactional(PGSTAT_KIND_SUBSCRIPTION, + InvalidOid, subid); } diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c index 17907e32789..5d3c184efb2 100644 --- a/src/backend/utils/activity/pgstat_xact.c +++ b/src/backend/utils/activity/pgstat_xact.c @@ -19,6 +19,18 @@ #include "utils/pgstat_internal.h" +typedef struct PgStat_PendingDroppedStatsItem +{ + xl_xact_stats_item item; + bool is_create; + dlist_node node; +} PgStat_PendingDroppedStatsItem; + + +static void AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit); +static void AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, + bool isCommit, int nestDepth); + static PgStat_SubXactStatus *pgStatXactStack = NULL; @@ -40,6 +52,7 @@ AtEOXact_PgStat(bool isCommit, bool parallel) Assert(xact_state->prev == NULL); AtEOXact_PgStat_Relations(xact_state, isCommit); + AtEOXact_PgStat_DroppedStats(xact_state, isCommit); } pgStatXactStack = NULL; @@ -47,6 +60,49 @@ AtEOXact_PgStat(bool isCommit, bool parallel) pgstat_clear_snapshot(); } +/* + * When committing, drop stats for objects dropped in the transaction. When + * aborting, drop stats for objects created in the transaction. + */ +static void +AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit) +{ + dlist_mutable_iter iter; + + if (xact_state->pending_drops_count == 0) + { + Assert(dlist_is_empty(&xact_state->pending_drops)); + return; + } + + dlist_foreach_modify(iter, &xact_state->pending_drops) + { + PgStat_PendingDroppedStatsItem *pending = + dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur); + + if (isCommit && !pending->is_create) + { + /* + * Transaction that dropped an object committed. Drop the stats + * too. + */ + /* will do work in subsequent commit */ + } + else if (!isCommit && pending->is_create) + { + /* + * Transaction that created an object aborted. Drop the stats + * associated with the object. + */ + /* will do work in subsequent commit */ + } + + dlist_delete(&pending->node); + xact_state->pending_drops_count--; + pfree(pending); + } +} + /* * Called from access/transam/xact.c at subtransaction commit/abort. */ @@ -64,11 +120,63 @@ AtEOSubXact_PgStat(bool isCommit, int nestDepth) pgStatXactStack = xact_state->prev; AtEOSubXact_PgStat_Relations(xact_state, isCommit, nestDepth); + AtEOSubXact_PgStat_DroppedStats(xact_state, isCommit, nestDepth); pfree(xact_state); } } +/* + * Like AtEOXact_PgStat_DroppedStats(), but for subtransactions. + */ +static void +AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, + bool isCommit, int nestDepth) +{ + PgStat_SubXactStatus *parent_xact_state; + dlist_mutable_iter iter; + + if (xact_state->pending_drops_count == 0) + return; + + parent_xact_state = pgstat_xact_stack_level_get(nestDepth - 1); + + dlist_foreach_modify(iter, &xact_state->pending_drops) + { + PgStat_PendingDroppedStatsItem *pending = + dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur); + + dlist_delete(&pending->node); + xact_state->pending_drops_count--; + + if (!isCommit && pending->is_create) + { + /* + * Subtransaction creating a new stats object aborted. Drop the + * stats object. + */ + /* will do work in subsequent commit */ + pfree(pending); + } + else if (isCommit) + { + /* + * Subtransaction dropping a stats object committed. Can't yet + * remove the stats object, the surrounding transaction might + * still abort. Pass it on to the parent. + */ + dlist_push_tail(&parent_xact_state->pending_drops, &pending->node); + parent_xact_state->pending_drops_count++; + } + else + { + pfree(pending); + } + } + + Assert(xact_state->pending_drops_count == 0); +} + /* * Save the transactional stats state at 2PC transaction prepare. */ @@ -130,6 +238,8 @@ pgstat_xact_stack_level_get(int nest_level) xact_state = (PgStat_SubXactStatus *) MemoryContextAlloc(TopTransactionContext, sizeof(PgStat_SubXactStatus)); + dlist_init(&xact_state->pending_drops); + xact_state->pending_drops_count = 0; xact_state->nest_level = nest_level; xact_state->prev = pgStatXactStack; xact_state->first = NULL; @@ -137,3 +247,113 @@ pgstat_xact_stack_level_get(int nest_level) } return xact_state; } + +/* + * Get stat items that need to be dropped at commit / abort. + * + * When committing, stats for objects that have been dropped in the + * transaction are returned. When aborting, stats for newly created objects is + * returned. + * + * Used by COMMIT / ABORT and 2PC PREPARE processing when building their + * respective WAL records, to ensure stats are dropped in case of a crash / on + * standbys. + */ +int +pgstat_get_transactional_drops(bool isCommit, xl_xact_stats_item **items) +{ + PgStat_SubXactStatus *xact_state = pgStatXactStack; + int nitems = 0; + dlist_iter iter; + + if (xact_state == NULL) + return 0; + + /* + * We expect to be called for subtransaction abort (which logs a WAL + * record), but not for subtransaction commit (which doesn't). + */ + Assert(!isCommit || xact_state->nest_level == 1); + Assert(!isCommit || xact_state->prev == NULL); + + *items = palloc(xact_state->pending_drops_count + * sizeof(PgStat_PendingDroppedStatsItem)); + + dlist_foreach(iter, &xact_state->pending_drops) + { + PgStat_PendingDroppedStatsItem *pending = + dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur); + + if (isCommit && pending->is_create) + continue; + if (!isCommit && !pending->is_create) + continue; + + Assert(nitems < xact_state->pending_drops_count); + (*items)[nitems++] = pending->item; + } + + return nitems; +} + +/* + * Execute scheduled drops post-commit. Called from xact_redo_commit() / + * xact_redo_abort() during recovery, and from FinishPreparedTransaction() + * during normal 2PC COMMIT/ABORT PREPARED processing. + */ +void +pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_item *items, bool is_redo) +{ + if (ndrops == 0) + return; + + for (int i = 0; i < ndrops; i++) + { + /* will do work in subsequent commit */ + } +} + +static void +create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, Oid objoid, bool is_create) +{ + int nest_level = GetCurrentTransactionNestLevel(); + PgStat_SubXactStatus *xact_state; + PgStat_PendingDroppedStatsItem *drop = (PgStat_PendingDroppedStatsItem *) + MemoryContextAlloc(TopTransactionContext, sizeof(PgStat_PendingDroppedStatsItem)); + + xact_state = pgstat_xact_stack_level_get(nest_level); + + drop->is_create = is_create; + drop->item.kind = kind; + drop->item.dboid = dboid; + drop->item.objoid = objoid; + + dlist_push_tail(&xact_state->pending_drops, &drop->node); + xact_state->pending_drops_count++; +} + +/* + * Create a stats entry for a newly created database object in a transactional + * manner. + * + * I.e. if the current (sub-)transaction aborts, the stats entry will also be + * dropped. + */ +void +pgstat_create_transactional(PgStat_Kind kind, Oid dboid, Oid objoid) +{ + create_drop_transactional_internal(kind, dboid, objoid, /* create */ true); +} + +/* + * Drop a stats entry for a just dropped database object in a transactional + * manner. + * + * I.e. if the current (sub-)transaction aborts, the stats entry will stay + * alive. + */ +void +pgstat_drop_transactional(PgStat_Kind kind, Oid dboid, Oid objoid) +{ + create_drop_transactional_internal(kind, dboid, objoid, /* create */ false); +} diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 53b73087eb8..efb30600e0d 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1965,6 +1965,7 @@ PgStat_MsgTabstat PgStat_MsgTempFile PgStat_MsgVacuum PgStat_MsgWal +PgStat_PendingDroppedStatsItem PgStat_SLRUStats PgStat_StatDBEntry PgStat_StatFuncEntry @@ -3771,6 +3772,8 @@ xl_xact_parsed_commit xl_xact_parsed_prepare xl_xact_prepare xl_xact_relfilenodes +xl_xact_stats_item +xl_xact_stats_items xl_xact_subxacts xl_xact_twophase xl_xact_xinfo -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0011-pgstat-revise-replslot-API-in-preparation-for-sh.patch" From 09ecf3b99e4779136b0c060c56ef4a126cb7dc51 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sat, 2 Apr 2022 15:36:30 -0700 Subject: [PATCH v68 11/31] pgstat: revise replslot API in preparation for shared memory stats. --- src/include/pgstat.h | 5 +++-- src/include/replication/slot.h | 1 + src/backend/replication/logical/logical.c | 2 +- src/backend/replication/slot.c | 15 ++++++++++++++- src/backend/utils/activity/pgstat_replslot.c | 9 ++++++--- 5 files changed, 25 insertions(+), 7 deletions(-) diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 4e0692d8b2b..7d8cea48efa 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -1150,8 +1150,9 @@ extern PgStat_TableStatus *find_tabstat_entry(Oid rel_id); extern void pgstat_reset_replslot_counters(void); extern void pgstat_reset_replslot_counter(const char *name); -extern void pgstat_report_replslot(const PgStat_StatReplSlotEntry *repSlotStat); -extern void pgstat_create_replslot(const char *slotname); +struct ReplicationSlot; +extern void pgstat_report_replslot(struct ReplicationSlot *slot, const PgStat_StatReplSlotEntry *repSlotStat); +extern void pgstat_create_replslot(struct ReplicationSlot *slot); extern void pgstat_drop_replslot(const char *slotname); diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index 24b30210c3e..1ee63c4cf44 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -216,6 +216,7 @@ extern bool ReplicationSlotsCountDBSlots(Oid dboid, int *nslots, int *nactive); extern void ReplicationSlotsDropDBSlots(Oid dboid); extern bool InvalidateObsoleteReplicationSlots(XLogSegNo oldestSegno); extern ReplicationSlot *SearchNamedReplicationSlot(const char *name, bool need_lock); +extern int ReplicationSlotIndex(ReplicationSlot *slot); extern void ReplicationSlotNameForTablesync(Oid suboid, Oid relid, char *syncslotname, int szslot); extern void ReplicationSlotDropAtPubNode(WalReceiverConn *wrconn, char *slotname, bool missing_ok); diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index e1f14aeecb5..656ec8f5555 100644 --- a/src/backend/replication/logical/logical.c +++ b/src/backend/replication/logical/logical.c @@ -1921,7 +1921,7 @@ UpdateDecodingStats(LogicalDecodingContext *ctx) repSlotStat.total_txns = rb->totalTxns; repSlotStat.total_bytes = rb->totalBytes; - pgstat_report_replslot(&repSlotStat); + pgstat_report_replslot(ctx->slot, &repSlotStat); rb->spillTxns = 0; rb->spillCount = 0; diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index b02571e4dd8..5d3b3fe6db6 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -356,7 +356,7 @@ ReplicationSlotCreate(const char *name, bool db_specific, * ReplicationSlotAllocationLock. */ if (SlotIsLogical(slot)) - pgstat_create_replslot(NameStr(slot->data.name)); + pgstat_create_replslot(slot); /* * Now that the slot has been marked as in_use and active, it's safe to @@ -399,6 +399,19 @@ SearchNamedReplicationSlot(const char *name, bool need_lock) return slot; } +/* + * Return the index of the replication slot in + * ReplicationSlotCtl->replication_slots. + * + * This is mainly useful to have an efficient key for storing replication slot + * stats. + */ +int +ReplicationSlotIndex(ReplicationSlot *slot) +{ + return slot - ReplicationSlotCtl->replication_slots; +} + /* * Find a previously created slot and mark it as used by this process. * diff --git a/src/backend/utils/activity/pgstat_replslot.c b/src/backend/utils/activity/pgstat_replslot.c index 8d64ecd8aaf..b1e894f2b21 100644 --- a/src/backend/utils/activity/pgstat_replslot.c +++ b/src/backend/utils/activity/pgstat_replslot.c @@ -85,7 +85,7 @@ pgstat_reset_replslot_counter(const char *name) * Report replication slot statistics. */ void -pgstat_report_replslot(const PgStat_StatReplSlotEntry *repSlotStat) +pgstat_report_replslot(ReplicationSlot *slot, const PgStat_StatReplSlotEntry *repSlotStat) { PgStat_MsgReplSlot msg; @@ -109,14 +109,17 @@ pgstat_report_replslot(const PgStat_StatReplSlotEntry *repSlotStat) /* * Report replication slot creation. + * + * NB: This gets called with ReplicationSlotAllocationLock already held, be + * careful about calling back into slot.c. */ void -pgstat_create_replslot(const char *slotname) +pgstat_create_replslot(ReplicationSlot *slot) { PgStat_MsgReplSlot msg; pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_REPLSLOT); - namestrcpy(&msg.m_slotname, slotname); + namestrcpy(&msg.m_slotname, NameStr(slot->data.name)); msg.m_create = true; msg.m_drop = false; pgstat_send(&msg, sizeof(PgStat_MsgReplSlot)); -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0012-pgstat-rename-some-pgstat_send_-functions-to-pgs.patch" From 5127a1da779c097b727c403b5e3764e8169aff37 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sun, 3 Apr 2022 13:22:11 -0700 Subject: [PATCH v68 12/31] pgstat: rename some pgstat_send_* functions to pgstat_report_*. Author: Reviewed-By: Discussion: https://postgr.es/m/ Backpatch: --- src/include/pgstat.h | 8 ++++---- src/backend/postmaster/bgwriter.c | 2 +- src/backend/postmaster/checkpointer.c | 10 +++++----- src/backend/postmaster/pgarch.c | 4 ++-- src/backend/postmaster/pgstat.c | 2 +- src/backend/postmaster/walwriter.c | 6 +++--- src/backend/utils/activity/pgstat_archiver.c | 2 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- src/backend/utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_wal.c | 8 ++++---- 10 files changed, 23 insertions(+), 23 deletions(-) diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 7d8cea48efa..26dde96726f 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -1018,21 +1018,21 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void); * Functions in pgstat_archiver.c */ -extern void pgstat_send_archiver(const char *xlog, bool failed); +extern void pgstat_report_archiver(const char *xlog, bool failed); /* * Functions in pgstat_bgwriter.c */ -extern void pgstat_send_bgwriter(void); +extern void pgstat_report_bgwriter(void); /* * Functions in pgstat_checkpointer.c */ -extern void pgstat_send_checkpointer(void); +extern void pgstat_report_checkpointer(void); /* @@ -1187,7 +1187,7 @@ extern void pgstat_drop_subscription(Oid subid); * Functions in pgstat_wal.c */ -extern void pgstat_send_wal(bool force); +extern void pgstat_report_wal(bool force); /* diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 447596b2a8b..91e6f6ea18a 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -241,7 +241,7 @@ BackgroundWriterMain(void) can_hibernate = BgBufferSync(&wb_context); /* Report pending statistics to the cumulative stats system */ - pgstat_send_bgwriter(); + pgstat_report_bgwriter(); if (FirstCallSinceLastCheckpoint()) { diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index 80914aa891c..e733c70368a 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -493,8 +493,8 @@ CheckpointerMain(void) CheckArchiveTimeout(); /* Report pending statistics to the cumulative stats system */ - pgstat_send_checkpointer(); - pgstat_send_wal(true); + pgstat_report_checkpointer(); + pgstat_report_wal(true); /* * If any checkpoint flags have been set, redo the loop to handle the @@ -571,8 +571,8 @@ HandleCheckpointerInterrupts(void) */ PendingCheckpointerStats.m_requested_checkpoints++; ShutdownXLOG(0, 0); - pgstat_send_checkpointer(); - pgstat_send_wal(true); + pgstat_report_checkpointer(); + pgstat_report_wal(true); /* Normal exit from the checkpointer is here */ proc_exit(0); /* done */ @@ -715,7 +715,7 @@ CheckpointWriteDelay(int flags, double progress) CheckArchiveTimeout(); /* Report interim statistics to the cumulative stats system */ - pgstat_send_checkpointer(); + pgstat_report_checkpointer(); /* * This sleep used to be connected to bgwriter_delay, typically 200ms. diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c index adfa404f464..0c8ca29f73d 100644 --- a/src/backend/postmaster/pgarch.c +++ b/src/backend/postmaster/pgarch.c @@ -480,7 +480,7 @@ pgarch_ArchiverCopyLoop(void) * Tell the cumulative stats system about the WAL file that we * successfully archived */ - pgstat_send_archiver(xlog, false); + pgstat_report_archiver(xlog, false); break; /* out of inner retry loop */ } @@ -490,7 +490,7 @@ pgarch_ArchiverCopyLoop(void) * Tell the cumulative stats system about the WAL file that we * failed to archive */ - pgstat_send_archiver(xlog, true); + pgstat_report_archiver(xlog, true); if (++failures >= NUM_ARCHIVE_RETRIES) { diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index 2ce4ffd2b64..6f985145b49 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/postmaster/pgstat.c @@ -793,7 +793,7 @@ pgstat_report_stat(bool disconnect) pgstat_send_funcstats(); /* Send WAL statistics */ - pgstat_send_wal(true); + pgstat_report_wal(true); /* Finally send SLRU statistics */ pgstat_send_slru(); diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index e2c7eb78d69..77aebb244cb 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -258,7 +258,7 @@ WalWriterMain(void) left_till_hibernate--; /* report pending statistics to the cumulative stats system */ - pgstat_send_wal(false); + pgstat_report_wal(false); /* * Sleep until we are signaled or WalWriterDelay has elapsed. If we @@ -297,11 +297,11 @@ HandleWalWriterInterrupts(void) /* * Force reporting remaining WAL statistics at process exit. * - * Since pgstat_send_wal is invoked with 'force' is false in main loop + * Since pgstat_report_wal is invoked with 'force' is false in main loop * to avoid overloading the cumulative stats system, there may exist * unreported stats counters for the WAL writer. */ - pgstat_send_wal(true); + pgstat_report_wal(true); proc_exit(0); } diff --git a/src/backend/utils/activity/pgstat_archiver.c b/src/backend/utils/activity/pgstat_archiver.c index c829b91b60c..09bc12070da 100644 --- a/src/backend/utils/activity/pgstat_archiver.c +++ b/src/backend/utils/activity/pgstat_archiver.c @@ -25,7 +25,7 @@ * Report archiver statistics */ void -pgstat_send_archiver(const char *xlog, bool failed) +pgstat_report_archiver(const char *xlog, bool failed) { PgStat_MsgArchiver msg; diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index 7e7eae80372..dfea88eca10 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -32,7 +32,7 @@ PgStat_MsgBgWriter PendingBgWriterStats; * Report bgwriter statistics */ void -pgstat_send_bgwriter(void) +pgstat_report_bgwriter(void) { /* We assume this initializes to zeroes */ static const PgStat_MsgBgWriter all_zeroes; diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index dbb9d861a4b..3f4e2054f55 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -32,7 +32,7 @@ PgStat_MsgCheckpointer PendingCheckpointerStats; * Report checkpointer statistics */ void -pgstat_send_checkpointer(void) +pgstat_report_checkpointer(void) { /* We assume this initializes to zeroes */ static const PgStat_MsgCheckpointer all_zeroes; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index ad999887630..8680809deef 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -31,8 +31,8 @@ PgStat_MsgWal WalStats; /* * WAL usage counters saved from pgWALUsage at the previous call to - * pgstat_send_wal(). This is used to calculate how much WAL usage - * happens between pgstat_send_wal() calls, by subtracting + * pgstat_report_wal(). This is used to calculate how much WAL usage + * happens between pgstat_report_wal() calls, by subtracting * the previous counters from the current ones. */ static WalUsage prevWalUsage; @@ -45,7 +45,7 @@ static WalUsage prevWalUsage; * passed since last one was sent to reach PGSTAT_STAT_INTERVAL. */ void -pgstat_send_wal(bool force) +pgstat_report_wal(bool force) { static TimestampTz sendTime = 0; @@ -133,7 +133,7 @@ void pgstat_wal_initialize(void) { /* - * Initialize prevWalUsage with pgWalUsage so that pgstat_send_wal() can + * Initialize prevWalUsage with pgWalUsage so that pgstat_report_wal() can * calculate how much pgWalUsage counters are increased by subtracting * prevWalUsage from pgWalUsage. */ -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0013-pgstat-store-statistics-in-shared-memory.patch" From d664632e2284135a580d787bd92576841a7b9969 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sun, 3 Apr 2022 12:43:57 -0700 Subject: [PATCH v68 13/31] pgstat: store statistics in shared memory. TODO: - verify all AFIXMEs - Tests for stats dropping: - think about / document checkpointer race conditions - check what needs to be done for index (and matview, ...?) stats - consistent naming around replacing "stats collector" references. Don't like "activity statistics". - Document stats_fetch_consistency / adjust existing documentation around stats fetching Previously activity statistics is collected via sockets and shared among backends through files periodically. Such files reaches tens of megabytes and are created at most every 1 second and such large data is serialized by stats collector then de-serialized on every backend periodically. To evade that large cost, this patch places activity statistics data on shared memory. Each backend accumulates statistics numbers locally then tries to move them onto the shared statistics at every transaction end but with intervals not shorter than 10 seconds. Until 60 second has elapsed since the last flushing to shared stats, lock failure postpones stats flushing to try to alleviate lock contention that slows down transactions. Finally stats flush waits for locks so that shared statistics doesn't get stale. FIXME: bump PGSTAT_FILE_FORMAT_ID Author: Kyotaro Horiguchi Author: Andres Freund Author: Melanie Plageman Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de Discussion: https://postgr.es/m/20220308205351.2xcn6k4x5yivcxyd@alap3.anarazel.de Discussion: https://postgr.es/m/20210319235115.y3wz7hpnnrshdyv6@alap3.anarazel.de --- src/include/miscadmin.h | 2 +- src/include/pgstat.h | 680 +-- src/include/storage/lwlock.h | 3 + src/include/utils/pgstat_internal.h | 599 +- src/include/utils/rel.h | 1 + src/include/utils/timeout.h | 1 + src/include/utils/wait_event.h | 1 - src/backend/access/transam/xlog.c | 26 +- src/backend/commands/vacuum.c | 7 - src/backend/commands/vacuumparallel.c | 2 + src/backend/postmaster/autovacuum.c | 197 +- src/backend/postmaster/checkpointer.c | 20 +- src/backend/postmaster/interrupt.c | 5 +- src/backend/postmaster/pgstat.c | 4845 ++++------------- src/backend/postmaster/postmaster.c | 91 +- src/backend/replication/logical/logical.c | 1 - src/backend/replication/slot.c | 28 +- src/backend/storage/buffer/bufmgr.c | 8 +- src/backend/storage/ipc/ipci.c | 2 + src/backend/storage/lmgr/lwlock.c | 8 +- src/backend/tcop/postgres.c | 29 +- src/backend/utils/activity/Makefile | 1 + src/backend/utils/activity/pgstat_archiver.c | 89 +- src/backend/utils/activity/pgstat_bgwriter.c | 81 +- .../utils/activity/pgstat_checkpointer.c | 93 +- src/backend/utils/activity/pgstat_database.c | 327 +- src/backend/utils/activity/pgstat_function.c | 165 +- src/backend/utils/activity/pgstat_relation.c | 626 ++- src/backend/utils/activity/pgstat_replslot.c | 189 +- src/backend/utils/activity/pgstat_shmem.c | 998 ++++ src/backend/utils/activity/pgstat_slru.c | 160 +- .../utils/activity/pgstat_subscription.c | 86 +- src/backend/utils/activity/pgstat_wal.c | 168 +- src/backend/utils/activity/pgstat_xact.c | 39 +- src/backend/utils/activity/wait_event.c | 3 - src/backend/utils/adt/pgstatfuncs.c | 10 +- src/backend/utils/cache/relcache.c | 5 + src/backend/utils/init/globals.c | 1 + src/backend/utils/init/miscinit.c | 3 - src/backend/utils/init/postinit.c | 12 + src/backend/utils/misc/guc.c | 21 + src/backend/utils/misc/postgresql.conf.sample | 1 + contrib/test_decoding/expected/stats.out | 1 + contrib/test_decoding/sql/stats.sql | 1 + doc/src/sgml/monitoring.sgml | 19 +- src/test/regress/expected/stats.out | 8 + src/test/regress/sql/stats.sql | 10 + src/tools/pgindent/typedefs.list | 63 +- 48 files changed, 4311 insertions(+), 5425 deletions(-) create mode 100644 src/backend/utils/activity/pgstat_shmem.c diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index 0abc3ad5405..1d89b2759d2 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -94,6 +94,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; +extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; @@ -333,7 +334,6 @@ typedef enum BackendType B_WAL_SENDER, B_WAL_WRITER, B_ARCHIVER, - B_STATS_COLLECTOR, B_LOGGER, } BackendType; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 26dde96726f..b4d1be8a79d 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -14,10 +14,8 @@ #include "datatype/timestamp.h" #include "portability/instr_time.h" #include "postmaster/pgarch.h" /* for MAX_XFN_CHARS */ -#include "replication/logicalproto.h" #include "utils/backend_progress.h" /* for backward compatibility */ #include "utils/backend_status.h" /* for backward compatibility */ -#include "utils/hsearch.h" #include "utils/relcache.h" #include "utils/wait_event.h" /* for backward compatibility */ @@ -27,8 +25,8 @@ * ---------- */ #define PGSTAT_STAT_PERMANENT_DIRECTORY "pg_stat" -#define PGSTAT_STAT_PERMANENT_FILENAME "pg_stat/global.stat" -#define PGSTAT_STAT_PERMANENT_TMPFILE "pg_stat/global.tmp" +#define PGSTAT_STAT_PERMANENT_FILENAME "pg_stat/pgstat.stat" +#define PGSTAT_STAT_PERMANENT_TMPFILE "pg_stat/pgstat.tmp" /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" @@ -69,6 +67,13 @@ typedef enum TrackFunctionsLevel TRACK_FUNC_ALL } TrackFunctionsLevel; +typedef enum PgStat_FetchConsistency +{ + PGSTAT_FETCH_CONSISTENCY_NONE, + PGSTAT_FETCH_CONSISTENCY_CACHE, + PGSTAT_FETCH_CONSISTENCY_SNAPSHOT, +} PgStat_FetchConsistency; + /* Values to track the cause of session termination */ typedef enum SessionEndType { @@ -95,7 +100,7 @@ typedef int64 PgStat_Counter; * PgStat_FunctionCounts The actual per-function counts kept by a backend * * This struct should contain only actual event counters, because we memcmp - * it against zeroes to detect whether there are any counts to transmit. + * it against zeroes to detect whether there are any pending stats. * * Note that the time counters are in instr_time format here. We convert to * microseconds in PgStat_Counter format when flushing out pending statistics. @@ -109,12 +114,11 @@ typedef struct PgStat_FunctionCounts } PgStat_FunctionCounts; /* ---------- - * PgStat_BackendFunctionEntry Entry in backend's per-function hash table + * PgStat_BackendFunctionEntry Non-flushed function stats. * ---------- */ typedef struct PgStat_BackendFunctionEntry { - Oid f_id; PgStat_FunctionCounts f_counts; } PgStat_BackendFunctionEntry; @@ -134,13 +138,22 @@ typedef struct PgStat_FunctionCallUsage instr_time f_start; } PgStat_FunctionCallUsage; +/* ---------- + * PgStat_BackendSubEntry Non-flushed subscription stats. + * ---------- + */ +typedef struct PgStat_BackendSubEntry +{ + PgStat_Counter apply_error_count; + PgStat_Counter sync_error_count; +} PgStat_BackendSubEntry; + /* ---------- * PgStat_TableCounts The actual per-table counts kept by a backend * * This struct should contain only actual event counters, because we memcmp - * it against zeroes to detect whether there are any counts to transmit. - * It is a component of PgStat_TableStatus (within-backend state) and - * PgStat_TableEntry (the transmitted message format). + * it against zeroes to detect whether there are any stats updates to apply. + * It is a component of PgStat_TableStatus (within-backend state). * * Note: for a table, tuples_returned is the number of tuples successfully * fetched by heap_getnext, while tuples_fetched is the number of tuples @@ -197,6 +210,7 @@ typedef struct PgStat_TableStatus bool t_shared; /* is it a shared catalog? */ struct PgStat_TableXactStatus *trans; /* lowest subxact's counts */ PgStat_TableCounts t_counts; /* event counts to be sent */ + Relation relation; /* rel that is using this entry */ } PgStat_TableStatus; /* ---------- @@ -224,562 +238,7 @@ typedef struct PgStat_TableXactStatus /* ------------------------------------------------------------ - * Message formats follow - * ------------------------------------------------------------ - */ - -/* ---------- - * The types of backend -> collector messages - * ---------- - */ -typedef enum StatMsgType -{ - PGSTAT_MTYPE_DUMMY, - PGSTAT_MTYPE_INQUIRY, - PGSTAT_MTYPE_TABSTAT, - PGSTAT_MTYPE_TABPURGE, - PGSTAT_MTYPE_DROPDB, - PGSTAT_MTYPE_RESETCOUNTER, - PGSTAT_MTYPE_RESETSHAREDCOUNTER, - PGSTAT_MTYPE_RESETSINGLECOUNTER, - PGSTAT_MTYPE_RESETSLRUCOUNTER, - PGSTAT_MTYPE_RESETREPLSLOTCOUNTER, - PGSTAT_MTYPE_RESETSUBCOUNTER, - PGSTAT_MTYPE_AUTOVAC_START, - PGSTAT_MTYPE_VACUUM, - PGSTAT_MTYPE_ANALYZE, - PGSTAT_MTYPE_ARCHIVER, - PGSTAT_MTYPE_BGWRITER, - PGSTAT_MTYPE_CHECKPOINTER, - PGSTAT_MTYPE_WAL, - PGSTAT_MTYPE_SLRU, - PGSTAT_MTYPE_FUNCSTAT, - PGSTAT_MTYPE_FUNCPURGE, - PGSTAT_MTYPE_RECOVERYCONFLICT, - PGSTAT_MTYPE_TEMPFILE, - PGSTAT_MTYPE_DEADLOCK, - PGSTAT_MTYPE_CHECKSUMFAILURE, - PGSTAT_MTYPE_REPLSLOT, - PGSTAT_MTYPE_CONNECT, - PGSTAT_MTYPE_DISCONNECT, - PGSTAT_MTYPE_SUBSCRIPTIONDROP, - PGSTAT_MTYPE_SUBSCRIPTIONERROR, -} StatMsgType; - -/* ---------- - * PgStat_MsgHdr The common message header - * ---------- - */ -typedef struct PgStat_MsgHdr -{ - StatMsgType m_type; - int m_size; -} PgStat_MsgHdr; - -/* ---------- - * Space available in a message. This will keep the UDP packets below 1K, - * which should fit unfragmented into the MTU of the loopback interface. - * (Larger values of PGSTAT_MAX_MSG_SIZE would work for that on most - * platforms, but we're being conservative here.) - * ---------- - */ -#define PGSTAT_MAX_MSG_SIZE 1000 -#define PGSTAT_MSG_PAYLOAD (PGSTAT_MAX_MSG_SIZE - sizeof(PgStat_MsgHdr)) - - -/* ---------- - * PgStat_MsgDummy A dummy message, ignored by the collector - * ---------- - */ -typedef struct PgStat_MsgDummy -{ - PgStat_MsgHdr m_hdr; -} PgStat_MsgDummy; - -/* ---------- - * PgStat_MsgInquiry Sent by a backend to ask the collector - * to write the stats file(s). - * - * Ordinarily, an inquiry message prompts writing of the global stats file, - * the stats file for shared catalogs, and the stats file for the specified - * database. If databaseid is InvalidOid, only the first two are written. - * - * New file(s) will be written only if the existing file has a timestamp - * older than the specified cutoff_time; this prevents duplicated effort - * when multiple requests arrive at nearly the same time, assuming that - * backends send requests with cutoff_times a little bit in the past. - * - * clock_time should be the requestor's current local time; the collector - * uses this to check for the system clock going backward, but it has no - * effect unless that occurs. We assume clock_time >= cutoff_time, though. - * ---------- - */ -typedef struct PgStat_MsgInquiry -{ - PgStat_MsgHdr m_hdr; - TimestampTz clock_time; /* observed local clock time */ - TimestampTz cutoff_time; /* minimum acceptable file timestamp */ - Oid databaseid; /* requested DB (InvalidOid => shared only) */ -} PgStat_MsgInquiry; - -/* ---------- - * PgStat_TableEntry Per-table info in a MsgTabstat - * ---------- - */ -typedef struct PgStat_TableEntry -{ - Oid t_id; - PgStat_TableCounts t_counts; -} PgStat_TableEntry; - -/* ---------- - * PgStat_MsgTabstat Sent by the backend to report table - * and buffer access statistics. - * ---------- - */ -#define PGSTAT_NUM_TABENTRIES \ - ((PGSTAT_MSG_PAYLOAD - sizeof(Oid) - 3 * sizeof(int) - 5 * sizeof(PgStat_Counter)) \ - / sizeof(PgStat_TableEntry)) - -typedef struct PgStat_MsgTabstat -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; - int m_nentries; - int m_xact_commit; - int m_xact_rollback; - PgStat_Counter m_block_read_time; /* times in microseconds */ - PgStat_Counter m_block_write_time; - PgStat_Counter m_session_time; - PgStat_Counter m_active_time; - PgStat_Counter m_idle_in_xact_time; - PgStat_TableEntry m_entry[PGSTAT_NUM_TABENTRIES]; -} PgStat_MsgTabstat; - -/* ---------- - * PgStat_MsgTabpurge Sent by the backend to tell the collector - * about dead tables. - * ---------- - */ -#define PGSTAT_NUM_TABPURGE \ - ((PGSTAT_MSG_PAYLOAD - sizeof(Oid) - sizeof(int)) \ - / sizeof(Oid)) - -typedef struct PgStat_MsgTabpurge -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; - int m_nentries; - Oid m_tableid[PGSTAT_NUM_TABPURGE]; -} PgStat_MsgTabpurge; - -/* ---------- - * PgStat_MsgDropdb Sent by the backend to tell the collector - * about a dropped database - * ---------- - */ -typedef struct PgStat_MsgDropdb -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; -} PgStat_MsgDropdb; - -/* ---------- - * PgStat_MsgResetcounter Sent by the backend to tell the collector - * to reset counters - * ---------- - */ -typedef struct PgStat_MsgResetcounter -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; -} PgStat_MsgResetcounter; - -/* ---------- - * PgStat_MsgResetsharedcounter Sent by the backend to tell the collector - * to reset a shared counter - * ---------- - */ -typedef struct PgStat_MsgResetsharedcounter -{ - PgStat_MsgHdr m_hdr; - PgStat_Kind m_resettarget; -} PgStat_MsgResetsharedcounter; - -/* ---------- - * PgStat_MsgResetsinglecounter Sent by the backend to tell the collector - * to reset a single counter - * ---------- - */ -typedef struct PgStat_MsgResetsinglecounter -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; - PgStat_Kind m_resettype; - Oid m_objectid; -} PgStat_MsgResetsinglecounter; - -/* ---------- - * PgStat_MsgResetslrucounter Sent by the backend to tell the collector - * to reset a SLRU counter - * ---------- - */ -typedef struct PgStat_MsgResetslrucounter -{ - PgStat_MsgHdr m_hdr; - int m_index; -} PgStat_MsgResetslrucounter; - -/* ---------- - * PgStat_MsgResetreplslotcounter Sent by the backend to tell the collector - * to reset replication slot counter(s) - * ---------- - */ -typedef struct PgStat_MsgResetreplslotcounter -{ - PgStat_MsgHdr m_hdr; - NameData m_slotname; - bool clearall; -} PgStat_MsgResetreplslotcounter; - -/* ---------- - * PgStat_MsgResetsubcounter Sent by the backend to tell the collector - * to reset subscription counter(s) - * ---------- - */ -typedef struct PgStat_MsgResetsubcounter -{ - PgStat_MsgHdr m_hdr; - Oid m_subid; /* InvalidOid means reset all subscription - * stats */ -} PgStat_MsgResetsubcounter; - -/* ---------- - * PgStat_MsgAutovacStart Sent by the autovacuum daemon to signal - * that a database is going to be processed - * ---------- - */ -typedef struct PgStat_MsgAutovacStart -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; - TimestampTz m_start_time; -} PgStat_MsgAutovacStart; - -/* ---------- - * PgStat_MsgVacuum Sent by the backend or autovacuum daemon - * after VACUUM - * ---------- - */ -typedef struct PgStat_MsgVacuum -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; - Oid m_tableoid; - bool m_autovacuum; - TimestampTz m_vacuumtime; - PgStat_Counter m_live_tuples; - PgStat_Counter m_dead_tuples; -} PgStat_MsgVacuum; - -/* ---------- - * PgStat_MsgAnalyze Sent by the backend or autovacuum daemon - * after ANALYZE - * ---------- - */ -typedef struct PgStat_MsgAnalyze -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; - Oid m_tableoid; - bool m_autovacuum; - bool m_resetcounter; - TimestampTz m_analyzetime; - PgStat_Counter m_live_tuples; - PgStat_Counter m_dead_tuples; -} PgStat_MsgAnalyze; - -/* ---------- - * PgStat_MsgArchiver Sent by the archiver to update statistics. - * ---------- - */ -typedef struct PgStat_MsgArchiver -{ - PgStat_MsgHdr m_hdr; - bool m_failed; /* Failed attempt */ - char m_xlog[MAX_XFN_CHARS + 1]; - TimestampTz m_timestamp; -} PgStat_MsgArchiver; - -/* ---------- - * PgStat_MsgBgWriter Sent by the bgwriter to update statistics. - * ---------- - */ -typedef struct PgStat_MsgBgWriter -{ - PgStat_MsgHdr m_hdr; - - PgStat_Counter m_buf_written_clean; - PgStat_Counter m_maxwritten_clean; - PgStat_Counter m_buf_alloc; -} PgStat_MsgBgWriter; - -/* ---------- - * PgStat_MsgCheckpointer Sent by the checkpointer to update statistics. - * ---------- - */ -typedef struct PgStat_MsgCheckpointer -{ - PgStat_MsgHdr m_hdr; - - PgStat_Counter m_timed_checkpoints; - PgStat_Counter m_requested_checkpoints; - PgStat_Counter m_buf_written_checkpoints; - PgStat_Counter m_buf_written_backend; - PgStat_Counter m_buf_fsync_backend; - PgStat_Counter m_checkpoint_write_time; /* times in milliseconds */ - PgStat_Counter m_checkpoint_sync_time; -} PgStat_MsgCheckpointer; - -/* ---------- - * PgStat_MsgWal Sent by backends and background processes to update WAL statistics. - * ---------- - */ -typedef struct PgStat_MsgWal -{ - PgStat_MsgHdr m_hdr; - PgStat_Counter m_wal_records; - PgStat_Counter m_wal_fpi; - uint64 m_wal_bytes; - PgStat_Counter m_wal_buffers_full; - PgStat_Counter m_wal_write; - PgStat_Counter m_wal_sync; - PgStat_Counter m_wal_write_time; /* time spent writing wal records in - * microseconds */ - PgStat_Counter m_wal_sync_time; /* time spent syncing wal records in - * microseconds */ -} PgStat_MsgWal; - -/* ---------- - * PgStat_MsgSLRU Sent by a backend to update SLRU statistics. - * ---------- - */ -typedef struct PgStat_MsgSLRU -{ - PgStat_MsgHdr m_hdr; - PgStat_Counter m_index; - PgStat_Counter m_blocks_zeroed; - PgStat_Counter m_blocks_hit; - PgStat_Counter m_blocks_read; - PgStat_Counter m_blocks_written; - PgStat_Counter m_blocks_exists; - PgStat_Counter m_flush; - PgStat_Counter m_truncate; -} PgStat_MsgSLRU; - -/* ---------- - * PgStat_MsgReplSlot Sent by a backend or a wal sender to update replication - * slot statistics. - * ---------- - */ -typedef struct PgStat_MsgReplSlot -{ - PgStat_MsgHdr m_hdr; - NameData m_slotname; - bool m_create; - bool m_drop; - PgStat_Counter m_spill_txns; - PgStat_Counter m_spill_count; - PgStat_Counter m_spill_bytes; - PgStat_Counter m_stream_txns; - PgStat_Counter m_stream_count; - PgStat_Counter m_stream_bytes; - PgStat_Counter m_total_txns; - PgStat_Counter m_total_bytes; -} PgStat_MsgReplSlot; - -/* ---------- - * PgStat_MsgSubscriptionDrop Sent by the backend and autovacuum to tell the - * collector about the dead subscription. - * ---------- - */ -typedef struct PgStat_MsgSubscriptionDrop -{ - PgStat_MsgHdr m_hdr; - Oid m_subid; -} PgStat_MsgSubscriptionDrop; - -/* ---------- - * PgStat_MsgSubscriptionError Sent by the apply worker or the table sync - * worker to report an error on the subscription. - * ---------- - */ -typedef struct PgStat_MsgSubscriptionError -{ - PgStat_MsgHdr m_hdr; - - Oid m_subid; - bool m_is_apply_error; -} PgStat_MsgSubscriptionError; - -/* ---------- - * PgStat_MsgRecoveryConflict Sent by the backend upon recovery conflict - * ---------- - */ -typedef struct PgStat_MsgRecoveryConflict -{ - PgStat_MsgHdr m_hdr; - - Oid m_databaseid; - int m_reason; -} PgStat_MsgRecoveryConflict; - -/* ---------- - * PgStat_MsgTempFile Sent by the backend upon creating a temp file - * ---------- - */ -typedef struct PgStat_MsgTempFile -{ - PgStat_MsgHdr m_hdr; - - Oid m_databaseid; - size_t m_filesize; -} PgStat_MsgTempFile; - -/* ---------- - * PgStat_FunctionEntry Per-function info in a MsgFuncstat - * ---------- - */ -typedef struct PgStat_FunctionEntry -{ - Oid f_id; - PgStat_Counter f_numcalls; - PgStat_Counter f_total_time; /* times in microseconds */ - PgStat_Counter f_self_time; -} PgStat_FunctionEntry; - -/* ---------- - * PgStat_MsgFuncstat Sent by the backend to report function - * usage statistics. - * ---------- - */ -#define PGSTAT_NUM_FUNCENTRIES \ - ((PGSTAT_MSG_PAYLOAD - sizeof(Oid) - sizeof(int)) \ - / sizeof(PgStat_FunctionEntry)) - -typedef struct PgStat_MsgFuncstat -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; - int m_nentries; - PgStat_FunctionEntry m_entry[PGSTAT_NUM_FUNCENTRIES]; -} PgStat_MsgFuncstat; - -/* ---------- - * PgStat_MsgFuncpurge Sent by the backend to tell the collector - * about dead functions. - * ---------- - */ -#define PGSTAT_NUM_FUNCPURGE \ - ((PGSTAT_MSG_PAYLOAD - sizeof(Oid) - sizeof(int)) \ - / sizeof(Oid)) - -typedef struct PgStat_MsgFuncpurge -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; - int m_nentries; - Oid m_functionid[PGSTAT_NUM_FUNCPURGE]; -} PgStat_MsgFuncpurge; - -/* ---------- - * PgStat_MsgDeadlock Sent by the backend to tell the collector - * about a deadlock that occurred. - * ---------- - */ -typedef struct PgStat_MsgDeadlock -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; -} PgStat_MsgDeadlock; - -/* ---------- - * PgStat_MsgChecksumFailure Sent by the backend to tell the collector - * about checksum failures noticed. - * ---------- - */ -typedef struct PgStat_MsgChecksumFailure -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; - int m_failurecount; - TimestampTz m_failure_time; -} PgStat_MsgChecksumFailure; - -/* ---------- - * PgStat_MsgConnect Sent by the backend upon connection - * establishment - * ---------- - */ -typedef struct PgStat_MsgConnect -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; -} PgStat_MsgConnect; - -/* ---------- - * PgStat_MsgDisconnect Sent by the backend when disconnecting - * ---------- - */ -typedef struct PgStat_MsgDisconnect -{ - PgStat_MsgHdr m_hdr; - Oid m_databaseid; - SessionEndType m_cause; -} PgStat_MsgDisconnect; - -/* ---------- - * PgStat_Msg Union over all possible messages. - * ---------- - */ -typedef union PgStat_Msg -{ - PgStat_MsgHdr msg_hdr; - PgStat_MsgDummy msg_dummy; - PgStat_MsgInquiry msg_inquiry; - PgStat_MsgTabstat msg_tabstat; - PgStat_MsgTabpurge msg_tabpurge; - PgStat_MsgDropdb msg_dropdb; - PgStat_MsgResetcounter msg_resetcounter; - PgStat_MsgResetsharedcounter msg_resetsharedcounter; - PgStat_MsgResetsinglecounter msg_resetsinglecounter; - PgStat_MsgResetslrucounter msg_resetslrucounter; - PgStat_MsgResetreplslotcounter msg_resetreplslotcounter; - PgStat_MsgResetsubcounter msg_resetsubcounter; - PgStat_MsgAutovacStart msg_autovacuum_start; - PgStat_MsgVacuum msg_vacuum; - PgStat_MsgAnalyze msg_analyze; - PgStat_MsgArchiver msg_archiver; - PgStat_MsgBgWriter msg_bgwriter; - PgStat_MsgCheckpointer msg_checkpointer; - PgStat_MsgWal msg_wal; - PgStat_MsgSLRU msg_slru; - PgStat_MsgFuncstat msg_funcstat; - PgStat_MsgFuncpurge msg_funcpurge; - PgStat_MsgRecoveryConflict msg_recoveryconflict; - PgStat_MsgDeadlock msg_deadlock; - PgStat_MsgTempFile msg_tempfile; - PgStat_MsgChecksumFailure msg_checksumfailure; - PgStat_MsgReplSlot msg_replslot; - PgStat_MsgConnect msg_connect; - PgStat_MsgDisconnect msg_disconnect; - PgStat_MsgSubscriptionError msg_subscriptionerror; - PgStat_MsgSubscriptionDrop msg_subscriptiondrop; -} PgStat_Msg; - - -/* ------------------------------------------------------------ - * Statistic collector data structures follow + * Data structures on disk and in shared memory follow * * PGSTAT_FILE_FORMAT_ID should be changed whenever any of these * data structures change. @@ -811,7 +270,6 @@ typedef struct PgStat_BgWriterStats typedef struct PgStat_CheckpointerStats { - TimestampTz stats_timestamp; /* time of stats file update */ PgStat_Counter timed_checkpoints; PgStat_Counter requested_checkpoints; PgStat_Counter checkpoint_write_time; /* times in milliseconds */ @@ -823,7 +281,6 @@ typedef struct PgStat_CheckpointerStats typedef struct PgStat_StatDBEntry { - Oid databaseid; PgStat_Counter n_xact_commit; PgStat_Counter n_xact_rollback; PgStat_Counter n_blocks_fetched; @@ -855,34 +312,16 @@ typedef struct PgStat_StatDBEntry PgStat_Counter n_sessions_killed; TimestampTz stat_reset_timestamp; - TimestampTz stats_timestamp; /* time of db stats file update */ - - /* - * tables and functions must be last in the struct, because we don't write - * the pointers out to the stats file. - */ - HTAB *tables; - HTAB *functions; } PgStat_StatDBEntry; typedef struct PgStat_StatFuncEntry { - Oid functionid; - PgStat_Counter f_numcalls; PgStat_Counter f_total_time; /* times in microseconds */ PgStat_Counter f_self_time; } PgStat_StatFuncEntry; -typedef struct PgStat_GlobalStats -{ - TimestampTz stats_timestamp; /* time of stats file update */ - - PgStat_CheckpointerStats checkpointer; - PgStat_BgWriterStats bgwriter; -} PgStat_GlobalStats; - typedef struct PgStat_StatReplSlotEntry { NameData slotname; @@ -911,8 +350,6 @@ typedef struct PgStat_SLRUStats typedef struct PgStat_StatSubEntry { - Oid subid; /* hash key (must be first) */ - PgStat_Counter apply_error_count; PgStat_Counter sync_error_count; TimestampTz stat_reset_timestamp; @@ -920,8 +357,6 @@ typedef struct PgStat_StatSubEntry typedef struct PgStat_StatTabEntry { - Oid tableid; - PgStat_Counter numscans; PgStat_Counter tuples_returned; @@ -969,14 +404,13 @@ typedef struct PgStat_WalStats */ /* functions called from postmaster */ -extern void pgstat_init(void); -extern void pgstat_reset_all(void); -extern int pgstat_start(void); -extern void allow_immediate_pgstat_restart(void); +extern Size StatsShmemSize(void); +extern void StatsShmemInit(void); -#ifdef EXEC_BACKEND -extern void PgstatCollectorMain(int argc, char *argv[]) pg_attribute_noreturn(); -#endif +/* Functions called during server startup / shutdown */ +extern void pgstat_restore_stats(void); +extern void pgstat_discard_stats(void); +extern void pgstat_before_server_shutdown(int code, Datum arg); /* Functions for backend initialization */ extern void pgstat_initialize(void); @@ -992,33 +426,24 @@ extern int pgstat_get_transactional_drops(bool isCommit, struct xl_xact_stats_it extern void pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_item *items, bool is_redo); /* Functions called from backends */ -extern void pgstat_report_stat(bool force); -extern void pgstat_vacuum_stat(void); -extern void pgstat_ping(void); +extern long pgstat_report_stat(bool force); extern void pgstat_reset_counters(void); extern void pgstat_reset_single_counter(PgStat_Kind kind, Oid objectid); extern void pgstat_reset_shared_counters(PgStat_Kind kind); /* stats accessors */ -extern PgStat_ArchiverStats *pgstat_fetch_stat_archiver(void); -extern PgStat_BgWriterStats *pgstat_fetch_stat_bgwriter(void); -extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void); -extern PgStat_StatDBEntry *pgstat_fetch_stat_dbentry(Oid dbid); -extern PgStat_StatFuncEntry *pgstat_fetch_stat_funcentry(Oid funcid); -extern PgStat_GlobalStats *pgstat_fetch_global(void); -extern PgStat_StatReplSlotEntry *pgstat_fetch_replslot(NameData slotname); -extern PgStat_StatSubEntry *pgstat_fetch_stat_subscription(Oid subid); -extern PgStat_SLRUStats *pgstat_fetch_slru(void); -extern PgStat_StatTabEntry *pgstat_fetch_stat_tabentry(Oid relid); -extern PgStat_WalStats *pgstat_fetch_stat_wal(void); +extern TimestampTz pgstat_get_stat_snapshot_timestamp(bool *have_snapshot); +/* helpers */ +extern PgStat_Kind pgstat_kind_from_str(char *kind_str); /* * Functions in pgstat_archiver.c */ extern void pgstat_report_archiver(const char *xlog, bool failed); +extern PgStat_ArchiverStats *pgstat_fetch_stat_archiver(void); /* @@ -1026,6 +451,7 @@ extern void pgstat_report_archiver(const char *xlog, bool failed); */ extern void pgstat_report_bgwriter(void); +extern PgStat_BgWriterStats *pgstat_fetch_stat_bgwriter(void); /* @@ -1033,6 +459,7 @@ extern void pgstat_report_bgwriter(void); */ extern void pgstat_report_checkpointer(void); +extern PgStat_CheckpointerStats *pgstat_fetch_stat_checkpointer(void); /* @@ -1055,6 +482,7 @@ extern void pgstat_report_connect(Oid dboid); #define pgstat_count_conn_txn_idle_time(n) \ (pgStatTransactionIdleTime += (n)) +extern PgStat_StatDBEntry *pgstat_fetch_stat_dbentry(Oid dbid); /* * Functions in pgstat_function.c @@ -1069,6 +497,7 @@ extern void pgstat_init_function_usage(struct FunctionCallInfoBaseData *fcinfo, extern void pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize); +extern PgStat_StatFuncEntry *pgstat_fetch_stat_funcentry(Oid funcid); extern PgStat_BackendFunctionEntry *find_funcstat_entry(Oid func_id); @@ -1081,6 +510,8 @@ extern void pgstat_drop_relation(Relation rel); extern void pgstat_copy_relation_stats(Relation dstrel, Relation srcrel); extern void pgstat_relation_init(Relation rel); +extern void pgstat_relation_assoc(Relation rel); +extern void pgstat_relation_unlink(Relation rel); extern void pgstat_report_autovac(Oid dboid); extern void pgstat_report_vacuum(Oid tableoid, bool shared, @@ -1089,8 +520,14 @@ extern void pgstat_report_analyze(Relation rel, PgStat_Counter livetuples, PgStat_Counter deadtuples, bool resetcounter); +/* + * If stats are hasn't been prepared yet, call pgstat_relation_assoc() to do + * so. See its comment for why this is done separately from + * pgstat_relation_init(). + */ #define pgstat_relation_should_count(rel) \ - (likely((rel)->pgstat_info != NULL)) + (likely((rel)->pgstat_info != NULL) ? true : \ + ((rel)->pgstat_enabled ? pgstat_relation_assoc(rel), true : false)) /* nontransactional event counts are simple enough to inline */ @@ -1141,6 +578,9 @@ extern void pgstat_twophase_postcommit(TransactionId xid, uint16 info, extern void pgstat_twophase_postabort(TransactionId xid, uint16 info, void *recdata, uint32 len); +extern PgStat_StatTabEntry *pgstat_fetch_stat_tabentry(Oid relid); +extern PgStat_StatTabEntry *pgstat_fetch_stat_tabentry_ext(bool shared, + Oid relid); extern PgStat_TableStatus *find_tabstat_entry(Oid rel_id); @@ -1153,7 +593,9 @@ extern void pgstat_reset_replslot_counter(const char *name); struct ReplicationSlot; extern void pgstat_report_replslot(struct ReplicationSlot *slot, const PgStat_StatReplSlotEntry *repSlotStat); extern void pgstat_create_replslot(struct ReplicationSlot *slot); -extern void pgstat_drop_replslot(const char *slotname); +extern void pgstat_acquire_replslot(struct ReplicationSlot *slot); +extern void pgstat_drop_replslot(struct ReplicationSlot *slot); +extern PgStat_StatReplSlotEntry *pgstat_fetch_replslot(NameData slotname); /* @@ -1170,6 +612,7 @@ extern void pgstat_count_slru_flush(int slru_idx); extern void pgstat_count_slru_truncate(int slru_idx); extern const char *pgstat_slru_name(int slru_idx); extern int pgstat_slru_index(const char *name); +extern PgStat_SLRUStats *pgstat_fetch_slru(void); /* @@ -1181,6 +624,7 @@ extern void pgstat_reset_subscription_counter(Oid subid); extern void pgstat_report_subscription_error(Oid subid, bool is_apply_error); extern void pgstat_create_subscription(Oid subid); extern void pgstat_drop_subscription(Oid subid); +extern PgStat_StatSubEntry *pgstat_fetch_stat_subscription(Oid subid); /* @@ -1188,6 +632,7 @@ extern void pgstat_drop_subscription(Oid subid); */ extern void pgstat_report_wal(bool force); +extern PgStat_WalStats *pgstat_fetch_stat_wal(void); /* @@ -1197,6 +642,9 @@ extern void pgstat_report_wal(bool force); /* GUC parameters */ extern PGDLLIMPORT bool pgstat_track_counts; extern PGDLLIMPORT int pgstat_track_functions; +extern PGDLLIMPORT int pgstat_fetch_consistency; + +/* No longer used, but will be removed with GUC */ extern char *pgstat_stat_directory; extern char *pgstat_stat_tmpname; extern char *pgstat_stat_filename; @@ -1207,7 +655,7 @@ extern char *pgstat_stat_filename; */ /* updated directly by bgwriter and bufmgr */ -extern PgStat_MsgBgWriter PendingBgWriterStats; +extern PgStat_BgWriterStats PendingBgWriterStats; /* @@ -1218,7 +666,7 @@ extern PgStat_MsgBgWriter PendingBgWriterStats; * Checkpointer statistics counters are updated directly by checkpointer and * bufmgr. */ -extern PgStat_MsgCheckpointer PendingCheckpointerStats; +extern PgStat_CheckpointerStats PendingCheckpointerStats; /* @@ -1245,7 +693,7 @@ extern SessionEndType pgStatSessionEndCause; */ /* updated directly by backends and background processes */ -extern PgStat_MsgWal WalStats; +extern PgStat_WalStats PendingWalStats; #endif /* PGSTAT_H */ diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h index c3d5889d7b2..33eb4c10339 100644 --- a/src/include/storage/lwlock.h +++ b/src/include/storage/lwlock.h @@ -190,6 +190,9 @@ typedef enum BuiltinTrancheIds LWTRANCHE_SHARED_TIDBITMAP, LWTRANCHE_PARALLEL_APPEND, LWTRANCHE_PER_XACT_PREDICATE_LIST, + LWTRANCHE_PGSTATS_DSA, + LWTRANCHE_PGSTATS_HASH, + LWTRANCHE_PGSTATS_DATA, LWTRANCHE_FIRST_USER_DEFINED } BuiltinTrancheIds; diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 16e99afa885..7efa9d32a97 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -3,7 +3,7 @@ * * Definitions for the PostgreSQL cumulative statistics system that should * only be needed by files implementing statistics support (rather than ones - * reporting / querying stats). + * reporting stats). * * Copyright (c) 2001-2022, PostgreSQL Global Development Group * @@ -14,21 +14,133 @@ #define PGSTAT_INTERNAL_H +#include "lib/dshash.h" +#include "lib/ilist.h" #include "pgstat.h" +#include "storage/lwlock.h" +#include "utils/dsa.h" -#define PGSTAT_STAT_INTERVAL 500 /* Minimum time between stats file - * updates; in milliseconds. */ - -/* ---------- - * The initial size hints for the hash tables used in the collector. - * ---------- +/* + * Types related to shared memory storage of statistics. + * + * Per-object statistics are stored in the "shared stats" hashtable. That + * table's entries (PgStatShared_HashEntry) contain a pointer to the actual stats + * data for the object (the size of the stats data varies depending on the + * kind of stats). The table is keyed by PgStat_HashKey. + * + * Once a backend has a reference to a shared stats entry, it increments the + * entry's refcount. Even after stats data is dropped (e.g. due to a DROP + * TABLE), the entry itself can only be deleted once all references have been + * released. + * + * These refcounts, in combination with a backend local hashtable + * (pgStatEntryRefHash, with entries pointing to PgStat_EntryRef) in front of + * the shared hash table, mean that most stats work can happen without + * touching the shared hash table, reducing contention. + * + * Once there are pending stats updates for a table PgStat_EntryRef->pending + * is allocated to contain a working space for as-of-yet-unapplied stats + * updates. Once the stats are flushed, PgStat_EntryRef->pending is freed. + * + * Each stat kind in the shared hash table has a fixed member + * PgStatShared_Common as the first element. */ -#define PGSTAT_DB_HASH_SIZE 16 -#define PGSTAT_TAB_HASH_SIZE 512 -#define PGSTAT_FUNCTION_HASH_SIZE 512 -#define PGSTAT_SUBSCRIPTION_HASH_SIZE 32 -#define PGSTAT_REPLSLOT_HASH_SIZE 32 + +/* struct for shared statistics hash entry key. */ +typedef struct PgStat_HashKey +{ + PgStat_Kind kind; /* statistics entry kind */ + Oid dboid; /* database ID. InvalidOid for shared objects. */ + Oid objoid; /* object ID, either table or function. */ +} PgStat_HashKey; + +/* + * Shared statistics hash entry. Doesn't itself contain any stats, but points + * to them (with ->body). That allows the stats entries themselves to be of + * variable size. + */ +typedef struct PgStatShared_HashEntry +{ + PgStat_HashKey key; /* hash key */ + + /* + * If dropped is set, backends need to release their references so that + * the memory for the entry can be freed. No new references may be made + * once marked as dropped. + */ + bool dropped; + + /* + * Refcount managing lifetime of the entry itself (as opposed to the + * dshash entry pointing to it). The stats lifetime has to be separate + * from the hash table entry lifetime because we allow backends to point + * to a stats entry without holding a hash table lock (and some other + * reasons). + * + * As long as the entry is not dropped, 1 is added to the refcount + * representing that the entry should not be dropped. In addition each + * backend that has a reference to the entry needs to increment the + * refcount as long as it does. + * + * May only be incremented / decremented while holding at least a shared + * lock on the dshash partition containing the entry. It needs to be an + * atomic variable because multiple backends can increment the refcount + * with just a shared lock. + * + * When the refcount reaches 0 the entry needs to be freed. + */ + pg_atomic_uint32 refcount; + + /* + * Pointer to shared stats. The stats entry always starts with + * PgStatShared_Common, embedded in a larger struct containing the + * PgStat_Kind specific stats fields. + */ + dsa_pointer body; +} PgStatShared_HashEntry; + +/* + * Common header struct for PgStatShm_Stat*Entry. + */ +typedef struct PgStatShared_Common +{ + uint32 magic; /* just a validity cross-check */ + /* lock protecting stats contents (i.e. data following the header) */ + LWLock lock; +} PgStatShared_Common; + +/* + * A backend local reference to a shared stats entry. As long as at least one + * such reference exists, the shared stats entry will not be released. + * + * If there are pending stats update to the shared stats, these are stored in + * ->pending. + */ +typedef struct PgStat_EntryRef +{ + /* + * Pointer to the PgStatShared_HashEntry entry in the shared stats + * hashtable. + */ + PgStatShared_HashEntry *shared_entry; + + /* + * Pointer to the stats data (i.e. PgStatShared_HashEntry->body), resolved + * as a local pointer, to avoid repeated dsa_get_address() calls. + */ + PgStatShared_Common *shared_stats; + + /* + * Pending statistics data that will need to be flushed to shared memory + * stats eventually. Each stats kind utilizing pending data defines what + * format its pending data has and needs to provide a + * PgStat_KindInfo->flush_pending_cb callback to merge pending into shared + * stats. + */ + void *pending; + dlist_node pending_node; /* membership in pgStatPending list */ +} PgStat_EntryRef; /* @@ -64,10 +176,93 @@ typedef struct PgStat_SubXactStatus } PgStat_SubXactStatus; +/* + * Metadata for a specific kinds of statistics. + */ +typedef struct PgStat_KindInfo +{ + /* + * Does a fixed number of stats object exist for this kind of stats or not + * (e.g. tables). + */ + bool fixed_amount:1; + + /* + * Can stats of this kind be accessed from another database? Determines + * whether a stats object gets included in stats snapshots. + */ + bool accessed_across_databases:1; + + /* + * For variable number stats: Identified on-disk using a name, rather than + * PgStat_HashKey. Probably only needed for replication slot stats. + */ + bool named_on_disk:1; + + /* + * The size of an entry in the shared stats hash table (pointed to by + * PgStatShared_HashEntry->body). + */ + uint32 shared_size; + + /* + * The offset/size of the statistics inside the shared stats entry. This + * is used to e.g. avoid touching lwlocks when serializing / restoring + * stats snapshot serialized to / from disk respectively. + */ + uint32 shared_data_off; + uint32 shared_data_len; + + /* + * The size of the pending data for this kind. E.g. how large + * PgStat_EntryRef->pending is. Used for allocations. + * + * -1 signal that an entry of this kind should never have a pending entry. + */ + uint32 pending_size; + + /* + * For variable number stats: flush pending stats. + */ + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + + /* + * For variable number stats: delete pending stats. Optional. + */ + void (*delete_pending_cb) (PgStat_EntryRef *sr); + + /* + * For variable number stats: reset the reset timestamp. Optional. + */ + void (*reset_timestamp_cb) (PgStatShared_Common *header, TimestampTz ts); + + /* + * For global statistics: Reset All. + */ + void (*reset_all_cb) (TimestampTz now); + + /* + * For global statistics: Fetch a snapshot of appropriate global stats. + */ + void (*snapshot_cb) (void); + + /* + * For named_on_disk_stats: + */ + void (*to_serialized_name) (const PgStatShared_Common *header, NameData *name); + bool (*from_serialized_name) (const NameData *name, PgStat_HashKey *key); + + const char *const name; +} PgStat_KindInfo; + + /* * List of SLRU names that we keep stats for. There is no central registry of * SLRUs, so we use this fixed list instead. The "other" entry is used for * all SLRUs without an explicit entry (e.g. SLRUs in extensions). + * + * This is only defined here so that SLRU_NUM_ELEMENTS is known for later type + * definitions. */ static const char *const slru_names[] = { "CommitTs", @@ -83,33 +278,265 @@ static const char *const slru_names[] = { #define SLRU_NUM_ELEMENTS lengthof(slru_names) +/* ---------- + * Types and definitions for different kinds of fixed-amount stats. + * + * Single-writer stats use the changecount mechanism to achieve low-overhead + * writes - they're obviously performance critical than reads. Check the + * definition of struct PgBackendStatus for some explanation of the + * changecount mechanism. + * + * Because the obvious implementation of resetting single-writer stats isn't + * compatible with that (another backend needs to write), we don't scribble on + * shared stats while resetting. Instead, just record the current counter + * values in a copy of the stats data, which is protected by ->lock. See + * pgstat_fetch_stat_(archiver|bgwriter|checkpointer) for the reader side. + * + * The only exception to that is the the stat_reset_timestamp in these + * structs, which is protected by ->lock, because it has to be written by + * another backend while resetting + * ---------- + */ + +typedef struct PgStatShared_Archiver +{ + /* lock protects ->reset_offset as well as stats->stat_reset_timestamp */ + LWLock lock; + uint32 changecount; + PgStat_ArchiverStats stats; + PgStat_ArchiverStats reset_offset; +} PgStatShared_Archiver; + +typedef struct PgStatShared_BgWriter +{ + /* lock protects ->reset_offset as well as stats->stat_reset_timestamp */ + LWLock lock; + uint32 changecount; + PgStat_BgWriterStats stats; + PgStat_BgWriterStats reset_offset; +} PgStatShared_BgWriter; + +typedef struct PgStatShared_Checkpointer +{ + /* lock protects ->reset_offset as well as stats->stat_reset_timestamp */ + LWLock lock; + uint32 changecount; + PgStat_CheckpointerStats stats; + PgStat_CheckpointerStats reset_offset; +} PgStatShared_Checkpointer; + +typedef struct PgStatShared_SLRU +{ + /* lock protects ->stats */ + LWLock lock; + PgStat_SLRUStats stats[SLRU_NUM_ELEMENTS]; +} PgStatShared_SLRU; + +typedef struct PgStatShared_Wal +{ + /* lock protects ->stats */ + LWLock lock; + PgStat_WalStats stats; +} PgStatShared_Wal; + + + +/* ---------- + * Types and definitions for different kinds of variable-amount stats. + * + * Each struct has to start with PgStatShared_Common, containing information + * common across the different types of stats. Afterwards kind-specific data + * follows. + * ---------- + */ + +typedef struct PgStatShared_Database +{ + PgStatShared_Common header; + PgStat_StatDBEntry stats; +} PgStatShared_Database; + +typedef struct PgStatShared_Relation +{ + PgStatShared_Common header; + PgStat_StatTabEntry stats; +} PgStatShared_Relation; + +typedef struct PgStatShared_Function +{ + PgStatShared_Common header; + PgStat_StatFuncEntry stats; +} PgStatShared_Function; + +typedef struct PgStatShared_Subscription +{ + PgStatShared_Common header; + PgStat_StatSubEntry stats; +} PgStatShared_Subscription; + +typedef struct PgStatShared_ReplSlot +{ + PgStatShared_Common header; + PgStat_StatReplSlotEntry stats; +} PgStatShared_ReplSlot; + + +/* + * Central shared memory entry for the cumulative stats system. + * + * Fixed amount stats, the dynamics shared memory hash table for + * non-fixed-amount stats, as well as remaining bits and pieces are all + * reached from here. + */ +typedef struct PgStat_ShmemControl +{ + void *raw_dsa_area; + + /* + * Stats for objects for which a variable number exists are kept in this + * shared hash table. See comment above PgStat_Kind for details. + */ + dshash_table_handle hash_handle; /* shared dbstat hash */ + + /* Has the stats system already been shut down? Just a debugging check. */ + bool is_shutdown; + + /* + * Whenever the for a dropped stats entry could not be freed (because + * backends still have references), this is incremented, causing backends + * to run pgstat_gc_entry_refs(), allowing that memory to be reclaimed. + */ + pg_atomic_uint64 gc_count; + + /* + * Stats data for fixed amount stats. + */ + PgStatShared_Archiver archiver; + PgStatShared_BgWriter bgwriter; + PgStatShared_Checkpointer checkpointer; + PgStatShared_SLRU slru; + PgStatShared_Wal wal; +} PgStat_ShmemControl; + + +/* + * Cached statistics snapshot + */ +typedef struct PgStat_Snapshot +{ + PgStat_FetchConsistency mode; + + /* time at which snapshot was taken */ + TimestampTz snapshot_timestamp; + + bool global_valid[PGSTAT_NUM_KINDS]; + + PgStat_ArchiverStats archiver; + + PgStat_BgWriterStats bgwriter; + + PgStat_CheckpointerStats checkpointer; + + PgStat_SLRUStats slru[SLRU_NUM_ELEMENTS]; + + PgStat_WalStats wal; + + /* to free snapshot in bulk */ + MemoryContext context; + struct pgstat_snapshot_hash *stats; +} PgStat_Snapshot; + + +/* + * Collection of backend-local stats state. + */ +typedef struct PgStat_LocalState +{ + PgStat_ShmemControl *shmem; + dsa_area *dsa; + dshash_table *shared_hash; + + /* the current statistics snapshot */ + PgStat_Snapshot snapshot; +} PgStat_LocalState; + + +/* + * Inline functions defined further below. + */ + +static inline void changecount_before_write(uint32 *cc); +static inline void changecount_after_write(uint32 *cc); +static inline uint32 changecount_before_read(uint32 *cc); +static inline bool changecount_after_read(uint32 *cc, uint32 cc_before); + +static inline void pgstat_copy_changecounted_stats(void *dst, void *src, size_t len, + uint32 *cc); + + /* * Functions in pgstat.c */ -extern void pgstat_setheader(PgStat_MsgHdr *hdr, StatMsgType mtype); -extern void pgstat_send(void *msg, int len); +const PgStat_KindInfo *pgstat_kind_info_for(PgStat_Kind kind); + #ifdef USE_ASSERT_CHECKING extern void pgstat_assert_is_up(void); #else #define pgstat_assert_is_up() ((void)true) #endif +extern void pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref); +extern PgStat_EntryRef *pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, Oid objoid, bool *created_entry); +extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, Oid dboid, Oid objoid); + +extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, Oid objoid); +extern void pgstat_snapshot_global(PgStat_Kind kind); + + +/* + * Functions in pgstat_archiver.c + */ + +extern void pgstat_archiver_reset_all_cb(TimestampTz now); +extern void pgstat_archiver_snapshot_cb(void); + + +/* + * Functions in pgstat_bgwriter.c + */ + +extern void pgstat_bgwriter_reset_all_cb(TimestampTz now); +extern void pgstat_bgwriter_snapshot_cb(void); + + +/* + * Functions in pgstat_checkpointer.c + */ + +extern void pgstat_checkpointer_reset_all_cb(TimestampTz now); +extern void pgstat_checkpointer_snapshot_cb(void); + /* * Functions in pgstat_database.c */ -extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern void pgstat_report_disconnect(Oid dboid); -extern void pgstat_update_dbstats(PgStat_MsgTabstat *tsmsg, TimestampTz now); +extern void pgstat_update_dbstats(TimestampTz now); +extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); + +extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); +extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); /* * Functions in pgstat_function.c */ -extern void pgstat_send_funcstats(void); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); /* @@ -120,23 +547,70 @@ extern void AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isC extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, int nestDepth); extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern void pgstat_send_tabstats(TimestampTz now, bool disconnect); + +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); + + +/* + * Functions in pgstat_replslot.c + */ + +extern void pgstat_replslot_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); +extern void pgstat_replslot_to_serialized_name_cb(const PgStatShared_Common *tmp, NameData *name); +extern bool pgstat_replslot_from_serialized_name_cb(const NameData *name, PgStat_HashKey *key); + + +/* + * Functions in pgstat_shmem.c + */ + +extern void pgstat_attach_shmem(void); +extern void pgstat_detach_shmem(void); + +extern PgStat_EntryRef *pgstat_get_entry_ref(PgStat_Kind kind, Oid dboid, Oid objoid, + bool create, bool *found); +extern bool pgstat_lock_entry(PgStat_EntryRef *entry_ref, bool nowait); +extern void pgstat_unlock_entry(PgStat_EntryRef *entry_ref); +extern bool pgstat_drop_entry(PgStat_Kind kind, Oid dboid, Oid objoid); +extern PgStat_EntryRef *pgstat_get_entry_ref_locked(PgStat_Kind kind, Oid dboid, Oid objoid, + bool nowait); +extern void pgstat_reset_entry(PgStat_Kind kind, Oid dboid, Oid objoid); +extern void pgstat_reset_entries_of_kind(PgStat_Kind kind); +extern void pgstat_reset_matching_entries(bool (*do_reset) (PgStatShared_HashEntry *, Datum), Datum match_data); + +extern void pgstat_request_entry_refs_gc(void); +extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, + PgStatShared_HashEntry *shhashent); /* * Functions in pgstat_slru.c */ -extern void pgstat_send_slru(void); +extern bool pgstat_slru_flush(bool nowait); +extern void pgstat_slru_reset_all_cb(TimestampTz now); +extern void pgstat_slru_snapshot_cb(void); /* * Functions in pgstat_wal.c */ +extern bool pgstat_flush_wal(bool nowait); extern void pgstat_wal_initialize(void); extern bool pgstat_wal_pending(void); +extern void pgstat_wal_reset_all_cb(TimestampTz now); +extern void pgstat_wal_snapshot_cb(void); + + +/* + * Functions in pgstat_subscription.c + */ + +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); /* @@ -152,29 +626,100 @@ extern void pgstat_create_transactional(PgStat_Kind kind, Oid dboid, Oid objoid) * Variables in pgstat.c */ -extern pgsocket pgStatSock; +extern PgStat_LocalState pgStatLocal; /* - * Variables in pgstat_database.c + * Variables in pgstat_slru.c */ -extern int pgStatXactCommit; -extern int pgStatXactRollback; +extern bool have_slrustats; /* - * Variables in pgstat_functions.c + * Implementation of inline functions declared above. */ -extern bool have_function_stats; +/* + * Helpers for changecount manipulation. See comments around struct + * PgBackendStatus for details. + */ + +static inline void +changecount_before_write(uint32 *cc) +{ + Assert((*cc & 1) == 0); + + START_CRIT_SECTION(); + (*cc)++; + pg_write_barrier(); +} + +static inline void +changecount_after_write(uint32 *cc) +{ + Assert((*cc & 1) == 1); + + pg_write_barrier(); + + (*cc)++; + + END_CRIT_SECTION(); +} + +static inline uint32 +changecount_before_read(uint32 *cc) +{ + uint32 before_cc = *cc; + + CHECK_FOR_INTERRUPTS(); + + pg_read_barrier(); + + return before_cc; +} + +/* + * Returns true if the read succeeded, false if it needs to be repeated. + */ +static inline bool +changecount_after_read(uint32 *cc, uint32 before_cc) +{ + uint32 after_cc; + + pg_read_barrier(); + + after_cc = *cc; + + /* was a write in progress when we started? */ + if (before_cc & 1) + return false; + + /* did writes start and complete while we read? */ + return before_cc == after_cc; +} /* - * Variables in pgstat_relation.c + * helper function for PgStat_KindInfo->snapshot_cb + * PgStat_KindInfo->reset_all_cb callbacks. + * + * Copies out the specified memory area following change-count protocol. */ +static inline void +pgstat_copy_changecounted_stats(void *dst, void *src, size_t len, + uint32 *cc) +{ + uint32 cc_before; -extern bool have_relation_stats; + do + { + cc_before = changecount_before_read(cc); + + memcpy(dst, src, len); + } + while (!changecount_after_read(cc, cc_before)); +} #endif /* PGSTAT_INTERNAL_H */ diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 121dbbc9a96..eadbd009045 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -246,6 +246,7 @@ typedef struct RelationData */ Oid rd_toastoid; /* Real TOAST table's OID, or InvalidOid */ + bool pgstat_enabled; /* should relation stats be counted */ /* use "struct" here to avoid needing to include pgstat.h: */ struct PgStat_TableStatus *pgstat_info; /* statistics collection area */ } RelationData; diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 099f91c61da..c068986d09a 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -32,6 +32,7 @@ typedef enum TimeoutId STANDBY_LOCK_TIMEOUT, IDLE_IN_TRANSACTION_SESSION_TIMEOUT, IDLE_SESSION_TIMEOUT, + IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index d870c592632..b578e2ec757 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -42,7 +42,6 @@ typedef enum WAIT_EVENT_CHECKPOINTER_MAIN, WAIT_EVENT_LOGICAL_APPLY_MAIN, WAIT_EVENT_LOGICAL_LAUNCHER_MAIN, - WAIT_EVENT_PGSTAT_MAIN, WAIT_EVENT_RECOVERY_WAL_STREAM, WAIT_EVENT_SYSLOGGER_MAIN, WAIT_EVENT_WAL_RECEIVER_MAIN, diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 17a56152f17..be2cd1a3ea5 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1869,7 +1869,7 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) WriteRqst.Flush = 0; XLogWrite(WriteRqst, tli, false); LWLockRelease(WALWriteLock); - WalStats.m_wal_buffers_full++; + PendingWalStats.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); } /* Re-acquire WALBufMappingLock and retry */ @@ -2227,10 +2227,10 @@ XLogWrite(XLogwrtRqst WriteRqst, TimeLineID tli, bool flexible) INSTR_TIME_SET_CURRENT(duration); INSTR_TIME_SUBTRACT(duration, start); - WalStats.m_wal_write_time += INSTR_TIME_GET_MICROSEC(duration); + PendingWalStats.wal_write_time += INSTR_TIME_GET_MICROSEC(duration); } - WalStats.m_wal_write++; + PendingWalStats.wal_write++; if (written <= 0) { @@ -5208,9 +5208,12 @@ StartupXLOG(void) } /* - * Reset pgstat data, because it may be invalid after recovery. + * Reset pgstat data, because it may be invalid after recovery. It's + * safe to do this here, because postmaster will not yet have started + * any other processes. NB: This basically just skips loading the data + * from disk, see pgstat_restore_stats() call in clean-startup path. */ - pgstat_reset_all(); + pgstat_discard_stats(); /* Check that the GUCs used to generate the WAL allow recovery */ CheckRequiredParameterValues(); @@ -5302,8 +5305,13 @@ StartupXLOG(void) performedWalRecovery = true; } else + { performedWalRecovery = false; + /* AFIXME: better location? */ + pgstat_restore_stats(); + } + /* * Finish WAL recovery. */ @@ -6104,8 +6112,8 @@ LogCheckpointEnd(bool restartpoint) CheckpointStats.ckpt_sync_end_t); /* Accumulate checkpoint timing summary data, in milliseconds. */ - PendingCheckpointerStats.m_checkpoint_write_time += write_msecs; - PendingCheckpointerStats.m_checkpoint_sync_time += sync_msecs; + PendingCheckpointerStats.checkpoint_write_time += write_msecs; + PendingCheckpointerStats.checkpoint_sync_time += sync_msecs; /* * All of the published timing statistics are accounted for. Only @@ -8032,10 +8040,10 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli) INSTR_TIME_SET_CURRENT(duration); INSTR_TIME_SUBTRACT(duration, start); - WalStats.m_wal_sync_time += INSTR_TIME_GET_MICROSEC(duration); + PendingWalStats.wal_sync_time += INSTR_TIME_GET_MICROSEC(duration); } - WalStats.m_wal_sync++; + PendingWalStats.wal_sync++; } /* diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c index f5c92c3860e..6ae0813cfbe 100644 --- a/src/backend/commands/vacuum.c +++ b/src/backend/commands/vacuum.c @@ -351,13 +351,6 @@ vacuum(List *relations, VacuumParams *params, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("PROCESS_TOAST required with VACUUM FULL"))); - /* - * Send info about dead objects to the cumulative stats system, unless - * we are in autovacuum --- autovacuum.c does this for itself. - */ - if ((params->options & VACOPT_VACUUM) && !IsAutoVacuumWorkerProcess()) - pgstat_vacuum_stat(); - /* * Create special memory context for cross-transaction storage. * diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c index 974a29e7a96..6b4f742578e 100644 --- a/src/backend/commands/vacuumparallel.c +++ b/src/backend/commands/vacuumparallel.c @@ -28,6 +28,7 @@ #include "access/amapi.h" #include "access/table.h" +#include "access/xact.h" #include "catalog/index.h" #include "commands/vacuum.h" #include "optimizer/paths.h" @@ -35,6 +36,7 @@ #include "storage/bufmgr.h" #include "tcop/tcopprot.h" #include "utils/lsyscache.h" +#include "utils/rel.h" /* * DSM keys for parallel vacuum. Unlike other parallel execution code, since diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c index c6d30fa5271..f36c40e852f 100644 --- a/src/backend/postmaster/autovacuum.c +++ b/src/backend/postmaster/autovacuum.c @@ -44,11 +44,12 @@ * Note that there can be more than one worker in a database concurrently. * They will store the table they are currently vacuuming in shared memory, so * that other workers avoid being blocked waiting for the vacuum lock for that - * table. They will also reload the pgstats data just before vacuuming each - * table, to avoid vacuuming a table that was just finished being vacuumed by - * another worker and thus is no longer noted in shared memory. However, - * there is a window (caused by pgstat delay) on which a worker may choose a - * table that was already vacuumed; this is a bug in the current design. + * table. They will also fetch the last time the table was vacuumed from + * pgstats just before vacuuming each table, to avoid vacuuming a table that + * was just finished being vacuumed by another worker and thus is no longer + * noted in shared memory. However, there is a small window (due to not yet + * holding the relation lock) during which a worker may choose a table that was + * already vacuumed; this is a bug in the current design. * * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California @@ -129,9 +130,6 @@ int autovacuum_vac_cost_limit; int Log_autovacuum_min_duration = 600000; -/* how long to keep pgstat data in the launcher, in milliseconds */ -#define STATS_READ_DELAY 1000 - /* the minimum allowed time between two awakenings of the launcher */ #define MIN_AUTOVAC_SLEEPTIME 100.0 /* milliseconds */ #define MAX_AUTOVAC_SLEEPTIME 300 /* seconds */ @@ -342,15 +340,11 @@ static void autovacuum_do_vac_analyze(autovac_table *tab, BufferAccessStrategy bstrategy); static AutoVacOpts *extract_autovac_opts(HeapTuple tup, TupleDesc pg_class_desc); -static PgStat_StatTabEntry *get_pgstat_tabentry_relid(Oid relid, bool isshared, - PgStat_StatDBEntry *shared, - PgStat_StatDBEntry *dbentry); static void perform_work_item(AutoVacuumWorkItem *workitem); static void autovac_report_activity(autovac_table *tab); static void autovac_report_workitem(AutoVacuumWorkItem *workitem, const char *nspname, const char *relname); static void avl_sigusr2_handler(SIGNAL_ARGS); -static void autovac_refresh_stats(void); @@ -555,12 +549,6 @@ AutoVacLauncherMain(int argc, char *argv[]) DatabaseListCxt = NULL; dlist_init(&DatabaseList); - /* - * Make sure pgstat also considers our stat data as gone. Note: we - * mustn't use autovac_refresh_stats here. - */ - pgstat_clear_snapshot(); - /* Now we can allow interrupts again */ RESUME_INTERRUPTS(); @@ -611,6 +599,12 @@ AutoVacLauncherMain(int argc, char *argv[]) SetConfigOption("default_transaction_isolation", "read committed", PGC_SUSET, PGC_S_OVERRIDE); + /* + * Even when system is configured to use a different fetch consistency, + * for autovac we always want fresh stats. + */ + SetConfigOption("stats_fetch_consistency", "none", PGC_SUSET, PGC_S_OVERRIDE); + /* * In emergency mode, just start a worker (unless shutdown was requested) * and go away. @@ -963,9 +957,6 @@ rebuild_database_list(Oid newdb) HTAB *dbhash; dlist_iter iter; - /* use fresh stats */ - autovac_refresh_stats(); - newcxt = AllocSetContextCreate(AutovacMemCxt, "Autovacuum database list", ALLOCSET_DEFAULT_SIZES); @@ -1184,9 +1175,6 @@ do_start_worker(void) ALLOCSET_DEFAULT_SIZES); oldcxt = MemoryContextSwitchTo(tmpcxt); - /* use fresh stats */ - autovac_refresh_stats(); - /* Get a list of databases */ dblist = get_database_list(); @@ -1642,6 +1630,12 @@ AutoVacWorkerMain(int argc, char *argv[]) SetConfigOption("synchronous_commit", "local", PGC_SUSET, PGC_S_OVERRIDE); + /* + * Even when system is configured to use a different fetch consistency, + * for autovac we always want fresh stats. + */ + SetConfigOption("stats_fetch_consistency", "none", PGC_SUSET, PGC_S_OVERRIDE); + /* * Get the info about the database we're going to work on. */ @@ -1966,8 +1960,6 @@ do_autovacuum(void) HASHCTL ctl; HTAB *table_toast_map; ListCell *volatile cell; - PgStat_StatDBEntry *shared; - PgStat_StatDBEntry *dbentry; BufferAccessStrategy bstrategy; ScanKeyData key; TupleDesc pg_class_desc; @@ -1986,22 +1978,9 @@ do_autovacuum(void) ALLOCSET_DEFAULT_SIZES); MemoryContextSwitchTo(AutovacMemCxt); - /* - * may be NULL if we couldn't find an entry (only happens if we are - * forcing a vacuum for anti-wrap purposes). - */ - dbentry = pgstat_fetch_stat_dbentry(MyDatabaseId); - /* Start a transaction so our commands have one to play into. */ StartTransactionCommand(); - /* - * Clean up any dead statistics entries for this DB. We always want to do - * this exactly once per DB-processing cycle, even if we find nothing - * worth vacuuming in the database. - */ - pgstat_vacuum_stat(); - /* * Compute the multixact age for which freezing is urgent. This is * normally autovacuum_multixact_freeze_max_age, but may be less if we are @@ -2039,9 +2018,6 @@ do_autovacuum(void) /* StartTransactionCommand changed elsewhere */ MemoryContextSwitchTo(AutovacMemCxt); - /* The database hash where pgstat keeps shared relations */ - shared = pgstat_fetch_stat_dbentry(InvalidOid); - classRel = table_open(RelationRelationId, AccessShareLock); /* create a copy so we can use it after closing pg_class */ @@ -2119,8 +2095,8 @@ do_autovacuum(void) /* Fetch reloptions and the pgstat entry for this table */ relopts = extract_autovac_opts(tuple, pg_class_desc); - tabentry = get_pgstat_tabentry_relid(relid, classForm->relisshared, - shared, dbentry); + tabentry = pgstat_fetch_stat_tabentry_ext(classForm->relisshared, + relid); /* Check if it needs vacuum or analyze */ relation_needs_vacanalyze(relid, relopts, classForm, tabentry, @@ -2203,8 +2179,8 @@ do_autovacuum(void) } /* Fetch the pgstat entry for this table */ - tabentry = get_pgstat_tabentry_relid(relid, classForm->relisshared, - shared, dbentry); + tabentry = pgstat_fetch_stat_tabentry_ext(classForm->relisshared, + relid); relation_needs_vacanalyze(relid, relopts, classForm, tabentry, effective_multixact_freeze_max_age, @@ -2418,12 +2394,8 @@ do_autovacuum(void) /* * Check whether pgstat data still says we need to vacuum this table. * It could have changed if something else processed the table while - * we weren't looking. - * - * Note: we have a special case in pgstat code to ensure that the - * stats we read are as up-to-date as possible, to avoid the problem - * that somebody just finished vacuuming this table. The window to - * the race condition is not closed but it is very small. + * we weren't looking. This doesn't entirely close the race condition, + * but it is very small. */ MemoryContextSwitchTo(AutovacMemCxt); tab = table_recheck_autovac(relid, table_toast_map, pg_class_desc, @@ -2768,29 +2740,6 @@ extract_autovac_opts(HeapTuple tup, TupleDesc pg_class_desc) return av; } -/* - * get_pgstat_tabentry_relid - * - * Fetch the pgstat entry of a table, either local to a database or shared. - */ -static PgStat_StatTabEntry * -get_pgstat_tabentry_relid(Oid relid, bool isshared, PgStat_StatDBEntry *shared, - PgStat_StatDBEntry *dbentry) -{ - PgStat_StatTabEntry *tabentry = NULL; - - if (isshared) - { - if (PointerIsValid(shared)) - tabentry = hash_search(shared->tables, &relid, - HASH_FIND, NULL); - } - else if (PointerIsValid(dbentry)) - tabentry = hash_search(dbentry->tables, &relid, - HASH_FIND, NULL); - - return tabentry; -} /* * table_recheck_autovac @@ -2812,7 +2761,6 @@ table_recheck_autovac(Oid relid, HTAB *table_toast_map, autovac_table *tab = NULL; bool wraparound; AutoVacOpts *avopts; - static bool reuse_stats = false; /* fetch the relation's relcache entry */ classTup = SearchSysCacheCopy1(RELOID, ObjectIdGetDatum(relid)); @@ -2836,35 +2784,6 @@ table_recheck_autovac(Oid relid, HTAB *table_toast_map, avopts = &hentry->ar_reloptions; } - /* - * Reuse the stats to recheck whether a relation needs to be vacuumed or - * analyzed if it was reloaded before and has not been cleared yet. This - * is necessary to avoid frequent refresh of stats, especially when there - * are very large number of relations and the refresh can cause lots of - * overhead. - * - * If we determined that a relation needs to be vacuumed or analyzed, - * based on the old stats, we refresh stats and recheck the necessity - * again. Because a relation may have already been vacuumed or analyzed by - * someone since the last reload of stats. - */ - if (reuse_stats) - { - recheck_relation_needs_vacanalyze(relid, avopts, classForm, - effective_multixact_freeze_max_age, - &dovacuum, &doanalyze, &wraparound); - - /* Quick exit if a relation doesn't need to be vacuumed or analyzed */ - if (!doanalyze && !dovacuum) - { - heap_freetuple(classTup); - return NULL; - } - } - - /* Use fresh stats and recheck again */ - autovac_refresh_stats(); - recheck_relation_needs_vacanalyze(relid, avopts, classForm, effective_multixact_freeze_max_age, &dovacuum, &doanalyze, &wraparound); @@ -2962,21 +2881,6 @@ table_recheck_autovac(Oid relid, HTAB *table_toast_map, tab->at_dobalance = !(avopts && (avopts->vacuum_cost_limit > 0 || avopts->vacuum_cost_delay > 0)); - - /* - * When we decide to do vacuum or analyze, the existing stats cannot - * be reused in the next cycle because it's cleared at the end of - * vacuum or analyze (by AtEOXact_PgStat()). - */ - reuse_stats = false; - } - else - { - /* - * If neither vacuum nor analyze is necessary, the existing stats is - * not cleared and can be reused in the next cycle. - */ - reuse_stats = true; } heap_freetuple(classTup); @@ -3001,17 +2905,10 @@ recheck_relation_needs_vacanalyze(Oid relid, bool *wraparound) { PgStat_StatTabEntry *tabentry; - PgStat_StatDBEntry *shared = NULL; - PgStat_StatDBEntry *dbentry = NULL; - - if (classForm->relisshared) - shared = pgstat_fetch_stat_dbentry(InvalidOid); - else - dbentry = pgstat_fetch_stat_dbentry(MyDatabaseId); /* fetch the pgstat table entry */ - tabentry = get_pgstat_tabentry_relid(relid, classForm->relisshared, - shared, dbentry); + tabentry = pgstat_fetch_stat_tabentry_ext(classForm->relisshared, + relid); relation_needs_vacanalyze(relid, avopts, classForm, tabentry, effective_multixact_freeze_max_age, @@ -3169,11 +3066,11 @@ relation_needs_vacanalyze(Oid relid, } /* - * If we found the table in the stats hash, and autovacuum is currently - * enabled, make a threshold-based decision whether to vacuum and/or - * analyze. If autovacuum is currently disabled, we must be here for - * anti-wraparound vacuuming only, so don't vacuum (or analyze) anything - * that's not being forced. + * If we found stats for the table, and autovacuum is currently enabled, + * make a threshold-based decision whether to vacuum and/or analyze. If + * autovacuum is currently disabled, we must be here for anti-wraparound + * vacuuming only, so don't vacuum (or analyze) anything that's not being + * forced. */ if (PointerIsValid(tabentry) && AutoVacuumingActive()) { @@ -3472,35 +3369,3 @@ AutoVacuumShmemInit(void) else Assert(found); } - -/* - * autovac_refresh_stats - * Refresh pgstats data for an autovacuum process - * - * Cause the next pgstats read operation to obtain fresh data, but throttle - * such refreshing in the autovacuum launcher. This is mostly to avoid - * rereading the pgstats files too many times in quick succession when there - * are many databases. - * - * Note: we avoid throttling in the autovac worker, as it would be - * counterproductive in the recheck logic. - */ -static void -autovac_refresh_stats(void) -{ - if (IsAutoVacuumLauncherProcess()) - { - static TimestampTz last_read = 0; - TimestampTz current_time; - - current_time = GetCurrentTimestamp(); - - if (!TimestampDifferenceExceeds(last_read, current_time, - STATS_READ_DELAY)) - return; - - last_read = current_time; - } - - pgstat_clear_snapshot(); -} diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e733c70368a..3cfac82ac09 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -212,6 +212,16 @@ CheckpointerMain(void) */ last_checkpoint_time = last_xlog_switch_time = (pg_time_t) time(NULL); + /* + * Write out stats after shutdown. This needs to be called by exactly one + * process during a normal shutdown, and since checkpointer is shut down + * very late... + * + * XXX: Are there potential issues with walsenders reporting stats at a + * later time? + */ + before_shmem_exit(pgstat_before_server_shutdown, 0); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid @@ -358,7 +368,7 @@ CheckpointerMain(void) if (((volatile CheckpointerShmemStruct *) CheckpointerShmem)->ckpt_flags) { do_checkpoint = true; - PendingCheckpointerStats.m_requested_checkpoints++; + PendingCheckpointerStats.requested_checkpoints++; } /* @@ -372,7 +382,7 @@ CheckpointerMain(void) if (elapsed_secs >= CheckPointTimeout) { if (!do_checkpoint) - PendingCheckpointerStats.m_timed_checkpoints++; + PendingCheckpointerStats.timed_checkpoints++; do_checkpoint = true; flags |= CHECKPOINT_CAUSE_TIME; } @@ -569,7 +579,7 @@ HandleCheckpointerInterrupts(void) * updates the statistics, increment the checkpoint request and flush * out pending statistic. */ - PendingCheckpointerStats.m_requested_checkpoints++; + PendingCheckpointerStats.requested_checkpoints++; ShutdownXLOG(0, 0); pgstat_report_checkpointer(); pgstat_report_wal(true); @@ -1262,9 +1272,9 @@ AbsorbSyncRequests(void) LWLockAcquire(CheckpointerCommLock, LW_EXCLUSIVE); /* Transfer stats counts into pending pgstats message */ - PendingCheckpointerStats.m_buf_written_backend + PendingCheckpointerStats.buf_written_backend += CheckpointerShmem->num_backend_writes; - PendingCheckpointerStats.m_buf_fsync_backend + PendingCheckpointerStats.buf_fsync_backend += CheckpointerShmem->num_backend_fsync; CheckpointerShmem->num_backend_writes = 0; diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c index 3f412dad2e6..1aed2e2e99e 100644 --- a/src/backend/postmaster/interrupt.c +++ b/src/backend/postmaster/interrupt.c @@ -98,9 +98,8 @@ SignalHandlerForCrashExit(SIGNAL_ARGS) * shut down and exit. * * Typically, this handler would be used for SIGTERM, but some processes use - * other signals. In particular, the checkpointer exits on SIGUSR2, the - * stats collector on SIGQUIT, and the WAL writer exits on either SIGINT - * or SIGTERM. + * other signals. In particular, the checkpointer exits on SIGUSR2, and the + * WAL writer exits on either SIGINT or SIGTERM. * * ShutdownRequestPending should be checked at a convenient place within the * main loop, or else the main loop should call HandleMainLoopInterrupts. diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index 6f985145b49..650edc0c8dd 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/postmaster/pgstat.c @@ -1,74 +1,51 @@ /* ---------- * pgstat.c + * Infrastructure for the cumulative statistics system. * - * All the statistics collector stuff hacked up in one big, ugly file. + * Provides the infrastructure to collect and access cumulative statistics, + * e.g. per-table access statistics, of all backends in shared memory. * - * TODO: - Separate collector, postmaster and backend stuff - * into different files. + * Most statistics updates are first first accumulated locally in each process + * as pending entries, then later flushed to shared memory (just after commit, + * or by idle-timeout). * - * - Add some automatic call for pgstat vacuuming. + * To avoid congestion on the shared memory, shared stats are updated once per + * PGSTAT_MIN_INTERVAL (1000ms). If some pending entries remain unflushed due + * to lock contention, an initial retry is done after + * PGSTAT_RETRY_MIN_INTERVAL (1000ms) and then doubled at every retry. Finally + * we perform a blocking stats update after PGSTAT_MAX_INTERVAL (60000ms). * - * - Add a pgstat config column to pg_database, so this - * entire thing can be enabled/disabled on a per db basis. + * AFIXME: architecture explanation. * - * Copyright (c) 2001-2022, PostgreSQL Global Development Group * - * src/backend/postmaster/pgstat.c + * NB: Code for individual kinds of statistics belongs into pgstat_*.c + * whenever possible, not here. + * + * Copyright (c) 2001-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/postmaster/pgstat.c * ---------- */ #include "postgres.h" #include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#ifdef HAVE_SYS_SELECT_H -#include -#endif -#include "access/heapam.h" -#include "access/htup_details.h" -#include "access/tableam.h" #include "access/transam.h" #include "access/xact.h" -#include "catalog/catalog.h" -#include "catalog/pg_database.h" -#include "catalog/pg_proc.h" -#include "catalog/pg_subscription.h" -#include "common/ip.h" -#include "libpq/libpq.h" -#include "libpq/pqsignal.h" -#include "mb/pg_wchar.h" +#include "common/hashfn.h" +#include "lib/dshash.h" #include "miscadmin.h" #include "pgstat.h" -#include "postmaster/autovacuum.h" -#include "postmaster/fork_process.h" -#include "postmaster/interrupt.h" -#include "postmaster/postmaster.h" -#include "replication/slot.h" -#include "replication/walsender.h" -#include "storage/backendid.h" -#include "storage/dsm.h" +#include "port/atomics.h" #include "storage/fd.h" #include "storage/ipc.h" -#include "storage/latch.h" -#include "storage/lmgr.h" +#include "storage/lwlock.h" #include "storage/pg_shmem.h" -#include "storage/proc.h" -#include "storage/procsignal.h" -#include "utils/builtins.h" +#include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" -#include "utils/ps_status.h" -#include "utils/rel.h" -#include "utils/snapmgr.h" #include "utils/timestamp.h" @@ -77,24 +54,50 @@ * ---------- */ -#define PGSTAT_RETRY_DELAY 10 /* How long to wait between checks for a - * new file; in milliseconds. */ +#define PGSTAT_MIN_INTERVAL 1000 /* Minimum interval of stats data + * updates; in milliseconds. */ -#define PGSTAT_MAX_WAIT_TIME 10000 /* Maximum time to wait for a stats - * file update; in milliseconds. */ +#define PGSTAT_RETRY_MIN_INTERVAL 1000 /* Initial retry interval after + * PGSTAT_MIN_INTERVAL */ -#define PGSTAT_INQ_INTERVAL 640 /* How often to ping the collector for a - * new file; in milliseconds. */ +#define PGSTAT_MAX_INTERVAL 60000 /* Longest interval of stats data + * updates */ -#define PGSTAT_RESTART_INTERVAL 60 /* How often to attempt to restart a - * failed statistics collector; in - * seconds. */ -#define PGSTAT_POLL_LOOP_COUNT (PGSTAT_MAX_WAIT_TIME / PGSTAT_RETRY_DELAY) -#define PGSTAT_INQ_LOOP_COUNT (PGSTAT_INQ_INTERVAL / PGSTAT_RETRY_DELAY) +/* ---------- + * Initial size hints for the hash tables used in statistics. + * ---------- + */ -/* Minimum receive buffer size for the collector's socket. */ -#define PGSTAT_MIN_RCVBUF (100 * 1024) +#define PGSTAT_SNAPSHOT_HASH_SIZE 512 + + +/* hash table for statistics snapshots entry */ +typedef struct PgStat_SnapshotEntry +{ + PgStat_HashKey key; + char status; /* for simplehash use */ + void *data; /* the stats data itself */ +} PgStat_SnapshotEntry; + + +/* ---------- + * Backend-local Hash Table Definitions + * ---------- + */ + +/* for stats snapshot entries */ +#define SH_PREFIX pgstat_snapshot +#define SH_ELEMENT_TYPE PgStat_SnapshotEntry +#define SH_KEY_TYPE PgStat_HashKey +#define SH_KEY key +#define SH_HASH_KEY(tb, key) \ + hash_bytes((unsigned char *)&key, sizeof(PgStat_HashKey)) +#define SH_EQUAL(tb, a, b) (memcmp(&a, &b, sizeof(PgStat_HashKey)) == 0) +#define SH_SCOPE static inline +#define SH_DEFINE +#define SH_DECLARE +#include "lib/simplehash.h" /* ---------- @@ -102,63 +105,19 @@ * ---------- */ -#ifdef EXEC_BACKEND -static pid_t pgstat_forkexec(void); -#endif +static void pgstat_write_statsfile(void); +static void pgstat_read_statsfile(void); -NON_EXEC_STATIC void PgstatCollectorMain(int argc, char *argv[]) pg_attribute_noreturn(); +static void pgstat_reset_all_stats(TimestampTz ts); -static PgStat_StatDBEntry *pgstat_get_db_entry(Oid databaseid, bool create); -static PgStat_StatTabEntry *pgstat_get_tab_entry(PgStat_StatDBEntry *dbentry, - Oid tableoid, bool create); -static PgStat_StatSubEntry *pgstat_get_subscription_entry(Oid subid, bool create); -static void pgstat_reset_subscription(PgStat_StatSubEntry *subentry, TimestampTz ts); -static void pgstat_write_statsfiles(bool permanent, bool allDbs); -static void pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent); -static HTAB *pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep); -static void pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash, - bool permanent); -static void backend_read_statsfile(void); +static bool pgstat_flush_pending_entries(bool nowait); -static bool pgstat_write_statsfile_needed(void); -static bool pgstat_db_requested(Oid databaseid); +static void pgstat_prep_snapshot(void); +static void pgstat_build_snapshot(void); +static void pgstat_build_snapshot_global(PgStat_Kind kind); -static PgStat_StatReplSlotEntry *pgstat_get_replslot_entry(NameData name, bool create_it); -static void pgstat_reset_replslot(PgStat_StatReplSlotEntry *slotstats, TimestampTz ts); - -static HTAB *pgstat_collect_oids(Oid catalogid, AttrNumber anum_oid); - -static void pgstat_setup_memcxt(void); - -static void pgstat_recv_inquiry(PgStat_MsgInquiry *msg, int len); -static void pgstat_recv_tabstat(PgStat_MsgTabstat *msg, int len); -static void pgstat_recv_tabpurge(PgStat_MsgTabpurge *msg, int len); -static void pgstat_recv_dropdb(PgStat_MsgDropdb *msg, int len); -static void pgstat_recv_resetcounter(PgStat_MsgResetcounter *msg, int len); -static void pgstat_recv_resetsharedcounter(PgStat_MsgResetsharedcounter *msg, int len); -static void pgstat_recv_resetsinglecounter(PgStat_MsgResetsinglecounter *msg, int len); -static void pgstat_recv_resetslrucounter(PgStat_MsgResetslrucounter *msg, int len); -static void pgstat_recv_resetreplslotcounter(PgStat_MsgResetreplslotcounter *msg, int len); -static void pgstat_recv_resetsubcounter(PgStat_MsgResetsubcounter *msg, int len); -static void pgstat_recv_autovac(PgStat_MsgAutovacStart *msg, int len); -static void pgstat_recv_vacuum(PgStat_MsgVacuum *msg, int len); -static void pgstat_recv_analyze(PgStat_MsgAnalyze *msg, int len); -static void pgstat_recv_archiver(PgStat_MsgArchiver *msg, int len); -static void pgstat_recv_bgwriter(PgStat_MsgBgWriter *msg, int len); -static void pgstat_recv_checkpointer(PgStat_MsgCheckpointer *msg, int len); -static void pgstat_recv_wal(PgStat_MsgWal *msg, int len); -static void pgstat_recv_slru(PgStat_MsgSLRU *msg, int len); -static void pgstat_recv_funcstat(PgStat_MsgFuncstat *msg, int len); -static void pgstat_recv_funcpurge(PgStat_MsgFuncpurge *msg, int len); -static void pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int len); -static void pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len); -static void pgstat_recv_checksum_failure(PgStat_MsgChecksumFailure *msg, int len); -static void pgstat_recv_connect(PgStat_MsgConnect *msg, int len); -static void pgstat_recv_disconnect(PgStat_MsgDisconnect *msg, int len); -static void pgstat_recv_replslot(PgStat_MsgReplSlot *msg, int len); -static void pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len); -static void pgstat_recv_subscription_drop(PgStat_MsgSubscriptionDrop *msg, int len); -static void pgstat_recv_subscription_error(PgStat_MsgSubscriptionError *msg, int len); +static inline bool pgstat_kind_valid(int ikind); +static inline void *shared_stat_entry_data(PgStat_Kind kind, PgStatShared_Common *entry); /* ---------- @@ -167,6 +126,7 @@ static void pgstat_recv_subscription_error(PgStat_MsgSubscriptionError *msg, int */ bool pgstat_track_counts = false; +int pgstat_fetch_consistency = PGSTAT_FETCH_CONSISTENCY_NONE; /* ---------- @@ -175,53 +135,37 @@ bool pgstat_track_counts = false; */ char *pgstat_stat_directory = NULL; + +/* No longer used, but will be removed with GUC */ char *pgstat_stat_filename = NULL; char *pgstat_stat_tmpname = NULL; -/* ---------- - * state shared with pgstat_*.c - * ---------- - */ - -pgsocket pgStatSock = PGINVALID_SOCKET; - /* ---------- * Local data + * + * NB: There should be only variables related to stats infrastructure here, + * not for specific kinds of stats. * ---------- */ -static struct sockaddr_storage pgStatAddr; +/* + * Memory contexts containing the pgStatEntryRefHash table, the + * pgStatSharedRef entries, and pending data respectively. Mostly to make it + * easier to track / attribute memory usage. + */ -static time_t last_pgstat_start_time; - -static bool pgStatRunningInCollector = false; +static MemoryContext pgStatPendingContext = NULL; /* - * Info about current "snapshot" of stats file + * List of PgStat_EntryRefs with unflushed pending stats. + * + * Newly pending entries should only ever be added to the end of the list, + * otherwise pgstat_flush_pending_entries() might not see them immediately. */ -static MemoryContext pgStatLocalContext = NULL; -static HTAB *pgStatDBHash = NULL; +static dlist_head pgStatPending = DLIST_STATIC_INIT(pgStatPending); -/* - * Cluster wide statistics, kept in the stats collector. - * Contains statistics that are not collected per database - * or per table. - */ -static PgStat_ArchiverStats archiverStats; -static PgStat_GlobalStats globalStats; -static PgStat_WalStats walStats; -static PgStat_SLRUStats slruStats[SLRU_NUM_ELEMENTS]; -static HTAB *replSlotStatHash = NULL; -static HTAB *subscriptionStatHash = NULL; - -/* - * List of OIDs of databases we need to write out. If an entry is InvalidOid, - * it means to write only the shared-catalog stats ("DB 0"); otherwise, we - * will write both that DB's data and the shared stats. - */ -static List *pending_write_requests = NIL; /* * For assertions that check pgstat is not used before initialization / after @@ -233,455 +177,238 @@ static bool pgstat_is_shutdown = false; #endif +PgStat_LocalState pgStatLocal; + + +/* + * The different kinds of statistics. + * + * If reasonably possible, handling specific to one kind of stats should go + * through this abstraction, rather than making more of pgstat.c aware. + * + * See comments for struct PgStat_KindInfo for details about the individual + * fields. + * + * XXX: It'd be nicer to define this outside of this file. But there doesn't + * seem to be a great way of doing that, given the split across multiple + * files. + */ +static const PgStat_KindInfo pgstat_kind_infos[PGSTAT_NUM_KINDS] = { + + /* stats kinds with a variable number of stats */ + + [PGSTAT_KIND_DATABASE] = { + .name = "database", + + .fixed_amount = false, + /* so pg_stat_database entries can be seen in all databases */ + .accessed_across_databases = true, + + .shared_size = sizeof(PgStatShared_Database), + .shared_data_off = offsetof(PgStatShared_Database, stats), + .shared_data_len = sizeof(((PgStatShared_Database *) 0)->stats), + .pending_size = sizeof(PgStat_StatDBEntry), + + .flush_pending_cb = pgstat_database_flush_cb, + .reset_timestamp_cb = pgstat_database_reset_timestamp_cb, + }, + + [PGSTAT_KIND_RELATION] = { + .name = "relation", + + .fixed_amount = false, + + .shared_size = sizeof(PgStatShared_Relation), + .shared_data_off = offsetof(PgStatShared_Relation, stats), + .shared_data_len = sizeof(((PgStatShared_Relation *) 0)->stats), + .pending_size = sizeof(PgStat_TableStatus), + + .flush_pending_cb = pgstat_relation_flush_cb, + .delete_pending_cb = pgstat_relation_delete_pending_cb, + }, + + [PGSTAT_KIND_FUNCTION] = { + .name = "function", + + .fixed_amount = false, + + .shared_size = sizeof(PgStatShared_Function), + .shared_data_off = offsetof(PgStatShared_Function, stats), + .shared_data_len = sizeof(((PgStatShared_Function *) 0)->stats), + .pending_size = sizeof(PgStat_BackendFunctionEntry), + + .flush_pending_cb = pgstat_function_flush_cb, + }, + + [PGSTAT_KIND_REPLSLOT] = { + .name = "replslot", + + .fixed_amount = false, + + .accessed_across_databases = true, + .named_on_disk = true, + + .shared_size = sizeof(PgStatShared_ReplSlot), + .shared_data_off = offsetof(PgStatShared_ReplSlot, stats), + .shared_data_len = sizeof(((PgStatShared_ReplSlot *) 0)->stats), + .pending_size = sizeof(PgStat_StatReplSlotEntry), + + .reset_timestamp_cb = pgstat_replslot_reset_timestamp_cb, + .to_serialized_name = pgstat_replslot_to_serialized_name_cb, + .from_serialized_name = pgstat_replslot_from_serialized_name_cb, + }, + + [PGSTAT_KIND_SUBSCRIPTION] = { + .name = "subscription", + + .fixed_amount = false, + /* so pg_stat_subscription_stats entries can be seen in all databases */ + .accessed_across_databases = true, + + .shared_size = sizeof(PgStatShared_Subscription), + .shared_data_off = offsetof(PgStatShared_Subscription, stats), + .shared_data_len = sizeof(((PgStatShared_Subscription *) 0)->stats), + .pending_size = sizeof(PgStat_BackendSubEntry), + + .flush_pending_cb = pgstat_subscription_flush_cb, + .reset_timestamp_cb = pgstat_subscription_reset_timestamp_cb, + }, + + + /* stats we have a fixed amount of (mostly 1) */ + + [PGSTAT_KIND_ARCHIVER] = { + .name = "archiver", + + .fixed_amount = true, + + .reset_all_cb = pgstat_archiver_reset_all_cb, + .snapshot_cb = pgstat_archiver_snapshot_cb, + }, + + [PGSTAT_KIND_BGWRITER] = { + .name = "bgwriter", + + .fixed_amount = true, + + .reset_all_cb = pgstat_bgwriter_reset_all_cb, + .snapshot_cb = pgstat_bgwriter_snapshot_cb, + }, + + [PGSTAT_KIND_CHECKPOINTER] = { + .name = "checkpointer", + + .fixed_amount = true, + + .reset_all_cb = pgstat_checkpointer_reset_all_cb, + .snapshot_cb = pgstat_checkpointer_snapshot_cb, + }, + + [PGSTAT_KIND_SLRU] = { + .name = "slru", + + .fixed_amount = true, + + .reset_all_cb = pgstat_slru_reset_all_cb, + .snapshot_cb = pgstat_slru_snapshot_cb, + }, + + [PGSTAT_KIND_WAL] = { + .name = "wal", + + .fixed_amount = true, + + .reset_all_cb = pgstat_wal_reset_all_cb, + .snapshot_cb = pgstat_wal_snapshot_cb, + }, +}; + + /* ------------------------------------------------------------ - * Public functions called from postmaster follow + * Functions manging the state of the stats system for all backends. * ------------------------------------------------------------ */ /* - * Called from postmaster at startup. Create the resources required - * by the statistics collector process. If unable to do so, do not - * fail --- better to let the postmaster start with stats collection - * disabled. + * Read on-disk stats into memory at server start. + * + * Should only be called by the startup process or in single user mode. */ void -pgstat_init(void) +pgstat_restore_stats(void) +{ + pgstat_read_statsfile(); +} + +/* + * Remove the stats file. This is currently used only if WAL recovery is + * needed after a crash. + * + * Should only be called by the startup process or in single user mode. + */ +void +pgstat_discard_stats(void) { - socklen_t alen; - struct addrinfo *addrs = NULL, - *addr, - hints; int ret; - fd_set rset; - struct timeval tv; - char test_byte; - int sel_res; - int tries = 0; -#define TESTBYTEVAL ((char) 199) + /* NB: this needs to be done even in single user mode */ - /* - * This static assertion verifies that we didn't mess up the calculations - * involved in selecting maximum payload sizes for our UDP messages. - * Because the only consequence of overrunning PGSTAT_MAX_MSG_SIZE would - * be silent performance loss from fragmentation, it seems worth having a - * compile-time cross-check that we didn't. - */ - StaticAssertStmt(sizeof(PgStat_Msg) <= PGSTAT_MAX_MSG_SIZE, - "maximum stats message size exceeds PGSTAT_MAX_MSG_SIZE"); - - /* - * Create the UDP socket for sending and receiving statistic messages - */ - hints.ai_flags = AI_PASSIVE; - hints.ai_family = AF_UNSPEC; - hints.ai_socktype = SOCK_DGRAM; - hints.ai_protocol = 0; - hints.ai_addrlen = 0; - hints.ai_addr = NULL; - hints.ai_canonname = NULL; - hints.ai_next = NULL; - ret = pg_getaddrinfo_all("localhost", NULL, &hints, &addrs); - if (ret || !addrs) + ret = unlink(PGSTAT_STAT_PERMANENT_FILENAME); + if (ret != 0) { - ereport(LOG, - (errmsg("could not resolve \"localhost\": %s", - gai_strerror(ret)))); - goto startup_failed; - } - - /* - * On some platforms, pg_getaddrinfo_all() may return multiple addresses - * only one of which will actually work (eg, both IPv6 and IPv4 addresses - * when kernel will reject IPv6). Worse, the failure may occur at the - * bind() or perhaps even connect() stage. So we must loop through the - * results till we find a working combination. We will generate LOG - * messages, but no error, for bogus combinations. - */ - for (addr = addrs; addr; addr = addr->ai_next) - { -#ifdef HAVE_UNIX_SOCKETS - /* Ignore AF_UNIX sockets, if any are returned. */ - if (addr->ai_family == AF_UNIX) - continue; -#endif - - if (++tries > 1) - ereport(LOG, - (errmsg("trying another address for the statistics collector"))); - - /* - * Create the socket. - */ - if ((pgStatSock = socket(addr->ai_family, SOCK_DGRAM, 0)) == PGINVALID_SOCKET) - { - ereport(LOG, - (errcode_for_socket_access(), - errmsg("could not create socket for statistics collector: %m"))); - continue; - } - - /* - * Bind it to a kernel assigned port on localhost and get the assigned - * port via getsockname(). - */ - if (bind(pgStatSock, addr->ai_addr, addr->ai_addrlen) < 0) - { - ereport(LOG, - (errcode_for_socket_access(), - errmsg("could not bind socket for statistics collector: %m"))); - closesocket(pgStatSock); - pgStatSock = PGINVALID_SOCKET; - continue; - } - - alen = sizeof(pgStatAddr); - if (getsockname(pgStatSock, (struct sockaddr *) &pgStatAddr, &alen) < 0) - { - ereport(LOG, - (errcode_for_socket_access(), - errmsg("could not get address of socket for statistics collector: %m"))); - closesocket(pgStatSock); - pgStatSock = PGINVALID_SOCKET; - continue; - } - - /* - * Connect the socket to its own address. This saves a few cycles by - * not having to respecify the target address on every send. This also - * provides a kernel-level check that only packets from this same - * address will be received. - */ - if (connect(pgStatSock, (struct sockaddr *) &pgStatAddr, alen) < 0) - { - ereport(LOG, - (errcode_for_socket_access(), - errmsg("could not connect socket for statistics collector: %m"))); - closesocket(pgStatSock); - pgStatSock = PGINVALID_SOCKET; - continue; - } - - /* - * Try to send and receive a one-byte test message on the socket. This - * is to catch situations where the socket can be created but will not - * actually pass data (for instance, because kernel packet filtering - * rules prevent it). - */ - test_byte = TESTBYTEVAL; - -retry1: - if (send(pgStatSock, &test_byte, 1, 0) != 1) - { - if (errno == EINTR) - goto retry1; /* if interrupted, just retry */ - ereport(LOG, - (errcode_for_socket_access(), - errmsg("could not send test message on socket for statistics collector: %m"))); - closesocket(pgStatSock); - pgStatSock = PGINVALID_SOCKET; - continue; - } - - /* - * There could possibly be a little delay before the message can be - * received. We arbitrarily allow up to half a second before deciding - * it's broken. - */ - for (;;) /* need a loop to handle EINTR */ - { - FD_ZERO(&rset); - FD_SET(pgStatSock, &rset); - - tv.tv_sec = 0; - tv.tv_usec = 500000; - sel_res = select(pgStatSock + 1, &rset, NULL, NULL, &tv); - if (sel_res >= 0 || errno != EINTR) - break; - } - if (sel_res < 0) - { - ereport(LOG, - (errcode_for_socket_access(), - errmsg("select() failed in statistics collector: %m"))); - closesocket(pgStatSock); - pgStatSock = PGINVALID_SOCKET; - continue; - } - if (sel_res == 0 || !FD_ISSET(pgStatSock, &rset)) - { - /* - * This is the case we actually think is likely, so take pains to - * give a specific message for it. - * - * errno will not be set meaningfully here, so don't use it. - */ - ereport(LOG, - (errcode(ERRCODE_CONNECTION_FAILURE), - errmsg("test message did not get through on socket for statistics collector"))); - closesocket(pgStatSock); - pgStatSock = PGINVALID_SOCKET; - continue; - } - - test_byte++; /* just make sure variable is changed */ - -retry2: - if (recv(pgStatSock, &test_byte, 1, 0) != 1) - { - if (errno == EINTR) - goto retry2; /* if interrupted, just retry */ - ereport(LOG, - (errcode_for_socket_access(), - errmsg("could not receive test message on socket for statistics collector: %m"))); - closesocket(pgStatSock); - pgStatSock = PGINVALID_SOCKET; - continue; - } - - if (test_byte != TESTBYTEVAL) /* strictly paranoia ... */ - { - ereport(LOG, - (errcode(ERRCODE_INTERNAL_ERROR), - errmsg("incorrect test message transmission on socket for statistics collector"))); - closesocket(pgStatSock); - pgStatSock = PGINVALID_SOCKET; - continue; - } - - /* If we get here, we have a working socket */ - break; - } - - /* Did we find a working address? */ - if (!addr || pgStatSock == PGINVALID_SOCKET) - goto startup_failed; - - /* - * Set the socket to non-blocking IO. This ensures that if the collector - * falls behind, statistics messages will be discarded; backends won't - * block waiting to send messages to the collector. - */ - if (!pg_set_noblock(pgStatSock)) - { - ereport(LOG, - (errcode_for_socket_access(), - errmsg("could not set statistics collector socket to nonblocking mode: %m"))); - goto startup_failed; - } - - /* - * Try to ensure that the socket's receive buffer is at least - * PGSTAT_MIN_RCVBUF bytes, so that it won't easily overflow and lose - * data. Use of UDP protocol means that we are willing to lose data under - * heavy load, but we don't want it to happen just because of ridiculously - * small default buffer sizes (such as 8KB on older Windows versions). - */ - { - int old_rcvbuf; - int new_rcvbuf; - socklen_t rcvbufsize = sizeof(old_rcvbuf); - - if (getsockopt(pgStatSock, SOL_SOCKET, SO_RCVBUF, - (char *) &old_rcvbuf, &rcvbufsize) < 0) - { - ereport(LOG, - (errmsg("%s(%s) failed: %m", "getsockopt", "SO_RCVBUF"))); - /* if we can't get existing size, always try to set it */ - old_rcvbuf = 0; - } - - new_rcvbuf = PGSTAT_MIN_RCVBUF; - if (old_rcvbuf < new_rcvbuf) - { - if (setsockopt(pgStatSock, SOL_SOCKET, SO_RCVBUF, - (char *) &new_rcvbuf, sizeof(new_rcvbuf)) < 0) - ereport(LOG, - (errmsg("%s(%s) failed: %m", "setsockopt", "SO_RCVBUF"))); - } - } - - pg_freeaddrinfo_all(hints.ai_family, addrs); - - /* Now that we have a long-lived socket, tell fd.c about it. */ - ReserveExternalFD(); - - return; - -startup_failed: - ereport(LOG, - (errmsg("disabling statistics collector for lack of working socket"))); - - if (addrs) - pg_freeaddrinfo_all(hints.ai_family, addrs); - - if (pgStatSock != PGINVALID_SOCKET) - closesocket(pgStatSock); - pgStatSock = PGINVALID_SOCKET; - - /* - * Adjust GUC variables to suppress useless activity, and for debugging - * purposes (seeing track_counts off is a clue that we failed here). We - * use PGC_S_OVERRIDE because there is no point in trying to turn it back - * on from postgresql.conf without a restart. - */ - SetConfigOption("track_counts", "off", PGC_INTERNAL, PGC_S_OVERRIDE); -} - -/* - * subroutine for pgstat_reset_all - */ -static void -pgstat_reset_remove_files(const char *directory) -{ - DIR *dir; - struct dirent *entry; - char fname[MAXPGPATH * 2]; - - dir = AllocateDir(directory); - while ((entry = ReadDir(dir, directory)) != NULL) - { - int nchars; - Oid tmp_oid; - - /* - * Skip directory entries that don't match the file names we write. - * See get_dbstat_filename for the database-specific pattern. - */ - if (strncmp(entry->d_name, "global.", 7) == 0) - nchars = 7; + if (errno == ENOENT) + elog(DEBUG2, + "didn't need to unlink permanent stats file \"%s\" - didn't exist", + PGSTAT_STAT_PERMANENT_FILENAME); else - { - nchars = 0; - (void) sscanf(entry->d_name, "db_%u.%n", - &tmp_oid, &nchars); - if (nchars <= 0) - continue; - /* %u allows leading whitespace, so reject that */ - if (strchr("0123456789", entry->d_name[3]) == NULL) - continue; - } - - if (strcmp(entry->d_name + nchars, "tmp") != 0 && - strcmp(entry->d_name + nchars, "stat") != 0) - continue; - - snprintf(fname, sizeof(fname), "%s/%s", directory, - entry->d_name); - unlink(fname); - } - FreeDir(dir); -} - -/* - * Remove the stats files. This is currently used only if WAL - * recovery is needed after a crash. - */ -void -pgstat_reset_all(void) -{ - pgstat_reset_remove_files(pgstat_stat_directory); - pgstat_reset_remove_files(PGSTAT_STAT_PERMANENT_DIRECTORY); -} - -#ifdef EXEC_BACKEND - -/* - * Format up the arglist for, then fork and exec, statistics collector process - */ -static pid_t -pgstat_forkexec(void) -{ - char *av[10]; - int ac = 0; - - av[ac++] = "postgres"; - av[ac++] = "--forkcol"; - av[ac++] = NULL; /* filled in by postmaster_forkexec */ - - av[ac] = NULL; - Assert(ac < lengthof(av)); - - return postmaster_forkexec(ac, av); -} -#endif /* EXEC_BACKEND */ - - -/* - * Called from postmaster at startup or after an existing collector - * died. Attempt to fire up a fresh statistics collector. - * - * Returns PID of child process, or 0 if fail. - * - * Note: if fail, we will be called again from the postmaster main loop. - */ -int -pgstat_start(void) -{ - time_t curtime; - pid_t pgStatPid; - - /* - * Check that the socket is there, else pgstat_init failed and we can do - * nothing useful. - */ - if (pgStatSock == PGINVALID_SOCKET) - return 0; - - /* - * Do nothing if too soon since last collector start. This is a safety - * valve to protect against continuous respawn attempts if the collector - * is dying immediately at launch. Note that since we will be re-called - * from the postmaster main loop, we will get another chance later. - */ - curtime = time(NULL); - if ((unsigned int) (curtime - last_pgstat_start_time) < - (unsigned int) PGSTAT_RESTART_INTERVAL) - return 0; - last_pgstat_start_time = curtime; - - /* - * Okay, fork off the collector. - */ -#ifdef EXEC_BACKEND - switch ((pgStatPid = pgstat_forkexec())) -#else - switch ((pgStatPid = fork_process())) -#endif - { - case -1: ereport(LOG, - (errmsg("could not fork statistics collector: %m"))); - return 0; - -#ifndef EXEC_BACKEND - case 0: - /* in postmaster child ... */ - InitPostmasterChild(); - - /* Close the postmaster's sockets */ - ClosePostmasterPorts(false); - - /* Drop our connection to postmaster's shared memory, as well */ - dsm_detach_all(); - PGSharedMemoryDetach(); - - PgstatCollectorMain(0, NULL); - break; -#endif - - default: - return (int) pgStatPid; + (errcode_for_file_access(), + errmsg("could not unlink permanent statistics file \"%s\": %m", + PGSTAT_STAT_PERMANENT_FILENAME))); + } + else + { + ereport(DEBUG2, + (errcode_for_file_access(), + errmsg("unlinked permanent statistics file \"%s\": %m", + PGSTAT_STAT_PERMANENT_FILENAME))); } - - /* shouldn't get here */ - return 0; } +/* + * pgstat_before_server_shutdown() needs to be called by exactly one process + * during regular server shutdowns. Otherwise all stats will be lost. + * + * We currently only write out stats for proc_exit(0). We might want to change + * that at some point... But right now pgstat_discard_stats() would be called + * during the start after a disorderly shutdown, anyway. + */ void -allow_immediate_pgstat_restart(void) +pgstat_before_server_shutdown(int code, Datum arg) { - last_pgstat_start_time = 0; + Assert(pgStatLocal.shmem != NULL); + Assert(!pgStatLocal.shmem->is_shutdown); + + /* + * Stats should only be reported after pgstat_initialize() and before + * pgstat_shutdown(). This is a convenient point to catch most violations + * of this rule. + */ + Assert(pgstat_is_initialized && !pgstat_is_shutdown); + + /* flush out our own pending changes before writing out */ + pgstat_report_stat(true); + + /* + * Only write out file during normal shutdown. Don't even signal that + * we've shutdown during irregular shutdowns, because the shutdown + * sequence isn't coordinated to ensure this backend shuts down last. + */ + if (code == 0) + { + pgStatLocal.shmem->is_shutdown = true; + pgstat_write_statsfile(); + } } @@ -701,6 +428,7 @@ static void pgstat_shutdown_hook(int code, Datum arg) { Assert(!pgstat_is_shutdown); + Assert(IsUnderPostmaster || !IsPostmasterEnvironment); /* * If we got as far as discovering our own database ID, we can flush out @@ -709,7 +437,15 @@ pgstat_shutdown_hook(int code, Datum arg) * failed backend starts might never get counted.) */ if (OidIsValid(MyDatabaseId)) - pgstat_report_stat(true); + pgstat_report_disconnect(MyDatabaseId); + + pgstat_report_stat(true); + + /* there shouldn't be any pending changes left */ + Assert(dlist_is_empty(&pgStatPending)); + dlist_init(&pgStatPending); + + pgstat_detach_shmem(); #ifdef USE_ASSERT_CHECKING pgstat_is_shutdown = true; @@ -727,6 +463,8 @@ pgstat_initialize(void) { Assert(!pgstat_is_initialized); + pgstat_attach_shmem(); + pgstat_wal_initialize(); /* Set up a process-exit hook to clean up */ @@ -745,323 +483,139 @@ pgstat_initialize(void) /* * Must be called by processes that performs DML: tcop/postgres.c, logical - * receiver processes, SPI worker, etc. to send the so far collected - * per-table and function usage statistics to the collector. Note that this - * is called only when not within a transaction, so it is fair to use - * transaction stop time as an approximation of current time. + * receiver processes, SPI worker, etc. to apply the so far collected + * per-table and function usage statistics to the shared statistics hashes. * - * "disconnect" is "true" only for the last call before the backend - * exits. This makes sure that no data is lost and that interrupted - * sessions are reported correctly. + * Updates are applied not more frequent than the interval of + * PGSTAT_MIN_INTERVAL milliseconds. They are also postponed on lock + * failure if force is false and there's no pending updates longer than + * PGSTAT_MAX_INTERVAL milliseconds. Postponed updates are retried in + * succeeding calls of this function. + * + * Returns the time until the next timing when updates are applied in + * milliseconds if there are no updates held for more than + * PGSTAT_MIN_INTERVAL milliseconds. + * + * Note that this is called only when not within a transaction, so it is fair + * to use transaction stop time as an approximation of current time. */ -void -pgstat_report_stat(bool disconnect) +long +pgstat_report_stat(bool force) { - static TimestampTz last_report = 0; - + static TimestampTz next_flush = 0; + static TimestampTz pending_since = 0; + static long retry_interval = 0; + bool partial_flush; TimestampTz now; + bool nowait; pgstat_assert_is_up(); - /* - * Don't expend a clock check if nothing to do. - */ - if (!have_relation_stats && - pgStatXactCommit == 0 && pgStatXactRollback == 0 && - !pgstat_wal_pending() && - !have_function_stats && !disconnect) - return; + /* Don't expend a clock check if nothing to do */ + if (dlist_is_empty(&pgStatPending) && + !have_slrustats && + !pgstat_wal_pending()) + { + return 0; + } /* - * Don't send a message unless it's been at least PGSTAT_STAT_INTERVAL - * msec since we last sent one, or the backend is about to exit. + * There should never be stats to report once stats are shut down. Can't + * assert that before the checks above, as there is an unconditional + * pgstat_report_stat() call in pgstat_shutdown_hook() - which at least + * the process that ran pgstat_before_server_shutdown() will still call. */ + Assert(!pgStatLocal.shmem->is_shutdown); + now = GetCurrentTransactionStopTimestamp(); - if (!disconnect && - !TimestampDifferenceExceeds(last_report, now, PGSTAT_STAT_INTERVAL)) - return; - last_report = now; + if (!force) + { + /* + * Don't flush stats too frequently. Return the time to the next + * flush. + */ + if (now < next_flush) + { + /* Record the epoch time if retrying. */ + if (pending_since == 0) + pending_since = now; - if (disconnect) - pgstat_report_disconnect(MyDatabaseId); + return (next_flush - now) / 1000; + } - /* First, send relation statistics */ - pgstat_send_tabstats(now, disconnect); + /* But, don't keep pending updates longer than PGSTAT_MAX_INTERVAL. */ - /* Now, send function statistics */ - pgstat_send_funcstats(); + if (pending_since > 0 && + TimestampDifferenceExceeds(pending_since, now, PGSTAT_MAX_INTERVAL)) + force = true; + } - /* Send WAL statistics */ - pgstat_report_wal(true); + pgstat_update_dbstats(now); - /* Finally send SLRU statistics */ - pgstat_send_slru(); + /* don't wait for lock acquisition when !force */ + nowait = !force; + + partial_flush = false; + + /* flush database / relation / function / ... stats */ + partial_flush |= pgstat_flush_pending_entries(nowait); + + /* flush wal stats */ + partial_flush |= pgstat_flush_wal(nowait); + + /* flush SLRU stats */ + partial_flush |= pgstat_slru_flush(nowait); + + /* + * Some of the pending stats may have not been flushed due to lock + * contention. If we have such pending stats here, let the caller know + * the retry interval. + */ + if (partial_flush) + { + /* Retain the epoch time */ + if (pending_since == 0) + pending_since = now; + + /* The interval is doubled at every retry. */ + if (retry_interval == 0) + retry_interval = PGSTAT_RETRY_MIN_INTERVAL * 1000; + else + retry_interval = retry_interval * 2; + + /* + * Determine the next retry interval so as not to get shorter than the + * previous interval. + */ + if (!TimestampDifferenceExceeds(pending_since, + now + 2 * retry_interval, + PGSTAT_MAX_INTERVAL)) + next_flush = now + retry_interval; + else + { + next_flush = pending_since + PGSTAT_MAX_INTERVAL * 1000; + retry_interval = next_flush - now; + } + + return retry_interval / 1000; + } + + /* Set the next time to update stats */ + next_flush = now + PGSTAT_MIN_INTERVAL * 1000; + retry_interval = 0; + pending_since = 0; + + return 0; } /* - * Will tell the collector about objects he can get rid of. + * Only for use by pgstat_reset_counters() */ -void -pgstat_vacuum_stat(void) +static bool +match_db_entries(PgStatShared_HashEntry *entry, Datum match_data) { - HTAB *htab; - PgStat_MsgTabpurge msg; - PgStat_MsgFuncpurge f_msg; - HASH_SEQ_STATUS hstat; - PgStat_StatDBEntry *dbentry; - PgStat_StatTabEntry *tabentry; - PgStat_StatFuncEntry *funcentry; - int len; - - if (pgStatSock == PGINVALID_SOCKET) - return; - - /* - * If not done for this transaction, read the statistics collector stats - * file into some hash tables. - */ - backend_read_statsfile(); - - /* - * Read pg_database and make a list of OIDs of all existing databases - */ - htab = pgstat_collect_oids(DatabaseRelationId, Anum_pg_database_oid); - - /* - * Search the database hash table for dead databases and tell the - * collector to drop them. - */ - hash_seq_init(&hstat, pgStatDBHash); - while ((dbentry = (PgStat_StatDBEntry *) hash_seq_search(&hstat)) != NULL) - { - Oid dbid = dbentry->databaseid; - - CHECK_FOR_INTERRUPTS(); - - /* the DB entry for shared tables (with InvalidOid) is never dropped */ - if (OidIsValid(dbid) && - hash_search(htab, (void *) &dbid, HASH_FIND, NULL) == NULL) - pgstat_drop_database(dbid); - } - - /* Clean up */ - hash_destroy(htab); - - /* - * Search for all the dead replication slots in stats hashtable and tell - * the stats collector to drop them. - */ - if (replSlotStatHash) - { - PgStat_StatReplSlotEntry *slotentry; - - hash_seq_init(&hstat, replSlotStatHash); - while ((slotentry = (PgStat_StatReplSlotEntry *) hash_seq_search(&hstat)) != NULL) - { - CHECK_FOR_INTERRUPTS(); - - if (SearchNamedReplicationSlot(NameStr(slotentry->slotname), true) == NULL) - pgstat_drop_replslot(NameStr(slotentry->slotname)); - } - } - - /* - * Repeat the above steps for subscriptions, if subscription stats are - * being collected. - */ - if (subscriptionStatHash) - { - PgStat_StatSubEntry *subentry; - - /* - * Read pg_subscription and make a list of OIDs of all existing - * subscriptions. - */ - htab = pgstat_collect_oids(SubscriptionRelationId, Anum_pg_subscription_oid); - - hash_seq_init(&hstat, subscriptionStatHash); - while ((subentry = (PgStat_StatSubEntry *) hash_seq_search(&hstat)) != NULL) - { - CHECK_FOR_INTERRUPTS(); - - if (hash_search(htab, (void *) &(subentry->subid), HASH_FIND, NULL) == NULL) - pgstat_drop_subscription(subentry->subid); - } - - hash_destroy(htab); - } - - /* - * Lookup our own database entry; if not found, nothing more to do. - */ - dbentry = (PgStat_StatDBEntry *) hash_search(pgStatDBHash, - (void *) &MyDatabaseId, - HASH_FIND, NULL); - if (dbentry == NULL || dbentry->tables == NULL) - return; - - /* - * Similarly to above, make a list of all known relations in this DB. - */ - htab = pgstat_collect_oids(RelationRelationId, Anum_pg_class_oid); - - /* - * Initialize our messages table counter to zero - */ - msg.m_nentries = 0; - - /* - * Check for all tables listed in stats hashtable if they still exist. - */ - hash_seq_init(&hstat, dbentry->tables); - while ((tabentry = (PgStat_StatTabEntry *) hash_seq_search(&hstat)) != NULL) - { - Oid tabid = tabentry->tableid; - - CHECK_FOR_INTERRUPTS(); - - if (hash_search(htab, (void *) &tabid, HASH_FIND, NULL) != NULL) - continue; - - /* - * Not there, so add this table's Oid to the message - */ - msg.m_tableid[msg.m_nentries++] = tabid; - - /* - * If the message is full, send it out and reinitialize to empty - */ - if (msg.m_nentries >= PGSTAT_NUM_TABPURGE) - { - len = offsetof(PgStat_MsgTabpurge, m_tableid[0]) - + msg.m_nentries * sizeof(Oid); - - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_TABPURGE); - msg.m_databaseid = MyDatabaseId; - pgstat_send(&msg, len); - - msg.m_nentries = 0; - } - } - - /* - * Send the rest - */ - if (msg.m_nentries > 0) - { - len = offsetof(PgStat_MsgTabpurge, m_tableid[0]) - + msg.m_nentries * sizeof(Oid); - - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_TABPURGE); - msg.m_databaseid = MyDatabaseId; - pgstat_send(&msg, len); - } - - /* Clean up */ - hash_destroy(htab); - - /* - * Now repeat the above steps for functions. However, we needn't bother - * in the common case where no function stats are being collected. - */ - if (dbentry->functions != NULL && - hash_get_num_entries(dbentry->functions) > 0) - { - htab = pgstat_collect_oids(ProcedureRelationId, Anum_pg_proc_oid); - - pgstat_setheader(&f_msg.m_hdr, PGSTAT_MTYPE_FUNCPURGE); - f_msg.m_databaseid = MyDatabaseId; - f_msg.m_nentries = 0; - - hash_seq_init(&hstat, dbentry->functions); - while ((funcentry = (PgStat_StatFuncEntry *) hash_seq_search(&hstat)) != NULL) - { - Oid funcid = funcentry->functionid; - - CHECK_FOR_INTERRUPTS(); - - if (hash_search(htab, (void *) &funcid, HASH_FIND, NULL) != NULL) - continue; - - /* - * Not there, so add this function's Oid to the message - */ - f_msg.m_functionid[f_msg.m_nentries++] = funcid; - - /* - * If the message is full, send it out and reinitialize to empty - */ - if (f_msg.m_nentries >= PGSTAT_NUM_FUNCPURGE) - { - len = offsetof(PgStat_MsgFuncpurge, m_functionid[0]) - + f_msg.m_nentries * sizeof(Oid); - - pgstat_send(&f_msg, len); - - f_msg.m_nentries = 0; - } - } - - /* - * Send the rest - */ - if (f_msg.m_nentries > 0) - { - len = offsetof(PgStat_MsgFuncpurge, m_functionid[0]) - + f_msg.m_nentries * sizeof(Oid); - - pgstat_send(&f_msg, len); - } - - hash_destroy(htab); - } -} - -/* - * Collect the OIDs of all objects listed in the specified system catalog - * into a temporary hash table. Caller should hash_destroy the result - * when done with it. (However, we make the table in CurrentMemoryContext - * so that it will be freed properly in event of an error.) - */ -static HTAB * -pgstat_collect_oids(Oid catalogid, AttrNumber anum_oid) -{ - HTAB *htab; - HASHCTL hash_ctl; - Relation rel; - TableScanDesc scan; - HeapTuple tup; - Snapshot snapshot; - - hash_ctl.keysize = sizeof(Oid); - hash_ctl.entrysize = sizeof(Oid); - hash_ctl.hcxt = CurrentMemoryContext; - htab = hash_create("Temporary table of OIDs", - PGSTAT_TAB_HASH_SIZE, - &hash_ctl, - HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); - - rel = table_open(catalogid, AccessShareLock); - snapshot = RegisterSnapshot(GetLatestSnapshot()); - scan = table_beginscan(rel, snapshot, 0, NULL); - while ((tup = heap_getnext(scan, ForwardScanDirection)) != NULL) - { - Oid thisoid; - bool isnull; - - thisoid = heap_getattr(tup, anum_oid, RelationGetDescr(rel), &isnull); - Assert(!isnull); - - CHECK_FOR_INTERRUPTS(); - - (void) hash_search(htab, (void *) &thisoid, HASH_ENTER, NULL); - } - table_endscan(scan); - UnregisterSnapshot(snapshot); - table_close(rel, AccessShareLock); - - return htab; + return entry->key.dboid == DatumGetObjectId(MyDatabaseId); } /* @@ -1073,14 +627,8 @@ pgstat_collect_oids(Oid catalogid, AttrNumber anum_oid) void pgstat_reset_counters(void) { - PgStat_MsgResetcounter msg; - - if (pgStatSock == PGINVALID_SOCKET) - return; - - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETCOUNTER); - msg.m_databaseid = MyDatabaseId; - pgstat_send(&msg, sizeof(msg)); + pgstat_reset_matching_entries(match_db_entries, + ObjectIdGetDatum(MyDatabaseId)); } /* @@ -1092,17 +640,15 @@ pgstat_reset_counters(void) void pgstat_reset_single_counter(PgStat_Kind kind, Oid objoid) { - PgStat_MsgResetsinglecounter msg; + TimestampTz ts = GetCurrentTimestamp(); - if (pgStatSock == PGINVALID_SOCKET) - return; + Assert(!pgstat_kind_info_for(kind)->fixed_amount); - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSINGLECOUNTER); - msg.m_databaseid = MyDatabaseId; - msg.m_resettype = kind; - msg.m_objectid = objoid; + /* Set the reset timestamp for the whole database */ + pgstat_reset_database_timestamp(MyDatabaseId, ts); - pgstat_send(&msg, sizeof(msg)); + /* reset the "single counter" */ + pgstat_reset_entry(kind, MyDatabaseId, objoid); } /* @@ -1114,45 +660,19 @@ pgstat_reset_single_counter(PgStat_Kind kind, Oid objoid) void pgstat_reset_shared_counters(PgStat_Kind kind) { - PgStat_MsgResetsharedcounter msg; + const PgStat_KindInfo *kind_info = pgstat_kind_info_for(kind); + TimestampTz now = GetCurrentTimestamp(); - if (pgStatSock == PGINVALID_SOCKET) - return; + Assert(kind_info->fixed_amount); - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSHAREDCOUNTER); - msg.m_resettarget = kind; - pgstat_send(&msg, sizeof(msg)); + kind_info->reset_all_cb(now); } -/* - * Send some junk data to the collector to increase traffic. + +/* ------------------------------------------------------------ + * Fetching of stats + * ------------------------------------------------------------ */ -void -pgstat_ping(void) -{ - PgStat_MsgDummy msg; - - if (pgStatSock == PGINVALID_SOCKET) - return; - - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_DUMMY); - pgstat_send(&msg, sizeof(msg)); -} - -/* - * Notify collector that we need fresh data. - */ -static void -pgstat_send_inquiry(TimestampTz clock_time, TimestampTz cutoff_time, Oid databaseid) -{ - PgStat_MsgInquiry msg; - - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_INQUIRY); - msg.clock_time = clock_time; - msg.cutoff_time = cutoff_time; - msg.databaseid = databaseid; - pgstat_send(&msg, sizeof(msg)); -} /* * Discard any data collected in the current transaction. Any subsequent @@ -1166,15 +686,19 @@ pgstat_clear_snapshot(void) { pgstat_assert_is_up(); - /* Release memory, if any was allocated */ - if (pgStatLocalContext) - MemoryContextDelete(pgStatLocalContext); + memset(&pgStatLocal.snapshot.global_valid, 0, + sizeof(pgStatLocal.snapshot.global_valid)); + pgStatLocal.snapshot.stats = NULL; + pgStatLocal.snapshot.mode = PGSTAT_FETCH_CONSISTENCY_NONE; - /* Reset variables */ - pgStatLocalContext = NULL; - pgStatDBHash = NULL; - replSlotStatHash = NULL; - subscriptionStatHash = NULL; + /* Release memory, if any was allocated */ + if (pgStatLocal.snapshot.context) + { + MemoryContextDelete(pgStatLocal.snapshot.context); + + /* Reset variables */ + pgStatLocal.snapshot.context = NULL; + } /* * Historically the backend_status.c facilities lived in this file, and @@ -1184,204 +708,396 @@ pgstat_clear_snapshot(void) pgstat_clear_backend_activity_snapshot(); } -/* - * Support function for the SQL-callable pgstat* functions. Returns - * the collected statistics for one database or NULL. NULL doesn't mean - * that the database doesn't exist, just that there are no statistics, so the - * caller is better off to report ZERO instead. - */ -PgStat_StatDBEntry * -pgstat_fetch_stat_dbentry(Oid dbid) +void * +pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, Oid objoid) { - /* - * If not done for this transaction, read the statistics collector stats - * file into some hash tables. - */ - backend_read_statsfile(); + PgStat_HashKey key; + PgStat_EntryRef *entry_ref; + void *stats_data; + const PgStat_KindInfo *kind_info = pgstat_kind_info_for(kind); - /* - * Lookup the requested database; return NULL if not found - */ - return (PgStat_StatDBEntry *) hash_search(pgStatDBHash, - (void *) &dbid, - HASH_FIND, NULL); -} + /* should be called from backends */ + Assert(IsUnderPostmaster || !IsPostmasterEnvironment); + AssertArg(!kind_info->fixed_amount); -/* - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the global statistics struct. - */ -PgStat_GlobalStats * -pgstat_fetch_global(void) -{ - backend_read_statsfile(); + pgstat_prep_snapshot(); - return &globalStats; -} + key.kind = kind; + key.dboid = dboid; + key.objoid = objoid; -/* - * Support function for the SQL-callable pgstat* functions. Returns - * the collected statistics for one table or NULL. NULL doesn't mean - * that the table doesn't exist, just that there are no statistics, so the - * caller is better off to report ZERO instead. - */ -PgStat_StatTabEntry * -pgstat_fetch_stat_tabentry(Oid relid) -{ - Oid dbid; - PgStat_StatDBEntry *dbentry; - PgStat_StatTabEntry *tabentry; + /* if we need to build a full snapshot, do so */ + if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT) + pgstat_build_snapshot(); - /* - * If not done for this transaction, read the statistics collector stats - * file into some hash tables. - */ - backend_read_statsfile(); - - /* - * Lookup our database, then look in its table hash table. - */ - dbid = MyDatabaseId; - dbentry = (PgStat_StatDBEntry *) hash_search(pgStatDBHash, - (void *) &dbid, - HASH_FIND, NULL); - if (dbentry != NULL && dbentry->tables != NULL) + /* if caching is desired, look up in cache */ + if (pgstat_fetch_consistency > PGSTAT_FETCH_CONSISTENCY_NONE) { - tabentry = (PgStat_StatTabEntry *) hash_search(dbentry->tables, - (void *) &relid, - HASH_FIND, NULL); - if (tabentry) - return tabentry; + PgStat_SnapshotEntry *entry = NULL; + + entry = pgstat_snapshot_lookup(pgStatLocal.snapshot.stats, key); + + if (entry) + return entry->data; + + /* + * If we built a full snapshot and the key is not in + * pgStatLocal.snapshot.stats, there are no matching stats. + */ + if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT) + return NULL; + } + + pgStatLocal.snapshot.mode = pgstat_fetch_consistency; + + entry_ref = pgstat_get_entry_ref(kind, dboid, objoid, false, NULL); + + if (entry_ref == NULL || entry_ref->shared_entry->dropped) + { + /* create empty entry when using PGSTAT_FETCH_CONSISTENCY_CACHE */ + if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_CACHE) + { + PgStat_SnapshotEntry *entry = NULL; + bool found; + + entry = pgstat_snapshot_insert(pgStatLocal.snapshot.stats, key, &found); + Assert(!found); + entry->data = NULL; + } + return NULL; } /* - * If we didn't find it, maybe it's a shared table. + * Allocate in caller's context for PGSTAT_FETCH_CONSISTENCY_NONE, + * otherwise we could quickly end up with a fair bit of memory used due to + * repeated accesses. */ - dbid = InvalidOid; - dbentry = (PgStat_StatDBEntry *) hash_search(pgStatDBHash, - (void *) &dbid, - HASH_FIND, NULL); - if (dbentry != NULL && dbentry->tables != NULL) + if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_NONE) + stats_data = palloc(kind_info->shared_data_len); + else + stats_data = MemoryContextAlloc(pgStatLocal.snapshot.context, + kind_info->shared_data_len); + memcpy(stats_data, + shared_stat_entry_data(kind, entry_ref->shared_stats), + kind_info->shared_data_len); + + if (pgstat_fetch_consistency > PGSTAT_FETCH_CONSISTENCY_NONE) { - tabentry = (PgStat_StatTabEntry *) hash_search(dbentry->tables, - (void *) &relid, - HASH_FIND, NULL); - if (tabentry) - return tabentry; + PgStat_SnapshotEntry *entry = NULL; + bool found; + + entry = pgstat_snapshot_insert(pgStatLocal.snapshot.stats, key, &found); + entry->data = stats_data; } - return NULL; + return stats_data; } - /* - * Support function for the SQL-callable pgstat* functions. Returns - * the collected statistics for one function or NULL. + * If a stats snapshot has been taken, return the timestamp at which that was + * done, and set *have_snapshot to true. Otherwise *have_snapshot is set to + * false. */ -PgStat_StatFuncEntry * -pgstat_fetch_stat_funcentry(Oid func_id) +TimestampTz +pgstat_get_stat_snapshot_timestamp(bool *have_snapshot) { - PgStat_StatDBEntry *dbentry; - PgStat_StatFuncEntry *funcentry = NULL; - - /* load the stats file if needed */ - backend_read_statsfile(); - - /* Lookup our database, then find the requested function. */ - dbentry = pgstat_fetch_stat_dbentry(MyDatabaseId); - if (dbentry != NULL && dbentry->functions != NULL) + if (pgStatLocal.snapshot.mode == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT) { - funcentry = (PgStat_StatFuncEntry *) hash_search(dbentry->functions, - (void *) &func_id, - HASH_FIND, NULL); + *have_snapshot = true; + return pgStatLocal.snapshot.snapshot_timestamp; } - return funcentry; + *have_snapshot = false; + + return 0; } /* - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the archiver statistics struct. + * Ensure snapshot for a kind of global stats exist. + * + * Typically used by the pgstat_fetch_* functions for a kind of stats, before + * massaging the data into the desired format. */ -PgStat_ArchiverStats * -pgstat_fetch_stat_archiver(void) +void +pgstat_snapshot_global(PgStat_Kind kind) { - backend_read_statsfile(); + AssertArg(pgstat_kind_valid(kind)); + AssertArg(pgstat_kind_info_for(kind)->fixed_amount); - return &archiverStats; + if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT) + pgstat_build_snapshot(); + else + pgstat_build_snapshot_global(kind); + + Assert(pgStatLocal.snapshot.global_valid[kind]); +} + +static void +pgstat_prep_snapshot(void) +{ + if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_NONE || + pgStatLocal.snapshot.stats != NULL) + return; + + if (!pgStatLocal.snapshot.context) + pgStatLocal.snapshot.context = AllocSetContextCreate(TopMemoryContext, + "PgStat Snapshot", + ALLOCSET_SMALL_SIZES); + + pgStatLocal.snapshot.stats = + pgstat_snapshot_create(pgStatLocal.snapshot.context, + PGSTAT_SNAPSHOT_HASH_SIZE, + NULL); +} + +static void +pgstat_build_snapshot(void) +{ + dshash_seq_status hstat; + PgStatShared_HashEntry *p; + + /* should only be called when we need a snapshot */ + Assert(pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT); + + /* snapshot already built */ + if (pgStatLocal.snapshot.mode == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT) + return; + + pgstat_prep_snapshot(); + + Assert(pgStatLocal.snapshot.stats->members == 0); + + pgStatLocal.snapshot.snapshot_timestamp = GetCurrentTimestamp(); + + /* + * Snapshot all variable stats. + */ + dshash_seq_init(&hstat, pgStatLocal.shared_hash, false); + while ((p = dshash_seq_next(&hstat)) != NULL) + { + PgStat_Kind kind = p->key.kind; + const PgStat_KindInfo *kind_info = pgstat_kind_info_for(kind); + bool found; + PgStat_SnapshotEntry *entry; + PgStatShared_Common *stats_data; + + /* + * Check if the stats object should be included in the snapshot. + * Unless the stats kind can be accessed from all databases (e.g. + * database stats themselves), we only include stats for the current + * database or objects not associated with a database (e.g. shared + * relations). + */ + if (p->key.dboid != MyDatabaseId && + p->key.dboid != InvalidOid && + !kind_info->accessed_across_databases) + continue; + + if (p->dropped) + continue; + + Assert(pg_atomic_read_u32(&p->refcount) > 0); + + stats_data = dsa_get_address(pgStatLocal.dsa, p->body); + Assert(stats_data); + + entry = pgstat_snapshot_insert(pgStatLocal.snapshot.stats, p->key, &found); + Assert(!found); + + entry->data = MemoryContextAlloc(pgStatLocal.snapshot.context, + kind_info->shared_size); + memcpy(entry->data, + shared_stat_entry_data(kind, stats_data), + kind_info->shared_size); + } + dshash_seq_term(&hstat); + + /* + * Build snapshot of all global stats. + */ + for (int kind = PGSTAT_KIND_FIRST; kind <= PGSTAT_KIND_LAST; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_kind_info_for(kind); + + if (!kind_info->fixed_amount) + { + Assert(kind_info->snapshot_cb == NULL); + continue; + } + + pgstat_build_snapshot_global(kind); + } + + pgStatLocal.snapshot.mode = PGSTAT_FETCH_CONSISTENCY_SNAPSHOT; +} + +static void +pgstat_build_snapshot_global(PgStat_Kind kind) +{ + const PgStat_KindInfo *kind_info = pgstat_kind_info_for(kind); + + Assert(kind_info->fixed_amount); + Assert(kind_info->snapshot_cb != NULL); + + if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_NONE) + { + /* rebuild every time */ + pgStatLocal.snapshot.global_valid[kind] = false; + } + else if (pgStatLocal.snapshot.global_valid[kind]) + { + /* in snapshot mode we shouldn't get called again */ + Assert(pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_CACHE); + return; + } + + Assert(!pgStatLocal.snapshot.global_valid[kind]); + + kind_info->snapshot_cb(); + + Assert(!pgStatLocal.snapshot.global_valid[kind]); + pgStatLocal.snapshot.global_valid[kind] = true; +} + + +/* ------------------------------------------------------------ + * Backend-local pending stats infrastructure + * ------------------------------------------------------------ + */ + +/* + * Returns the appropriate PgStat_EntryRef, preparing it to receive pending + * stats if not already done. + */ +PgStat_EntryRef * +pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, Oid objoid, bool *created_entry) +{ + PgStat_EntryRef *entry_ref; + + /* need to be able to flush out */ + Assert(pgstat_kind_info_for(kind)->flush_pending_cb != NULL); + + if (unlikely(!pgStatPendingContext)) + { + pgStatPendingContext = + AllocSetContextCreate(CacheMemoryContext, + "PgStat Pending", + ALLOCSET_SMALL_SIZES); + } + + entry_ref = pgstat_get_entry_ref(kind, dboid, objoid, + true, created_entry); + + if (entry_ref->pending == NULL) + { + size_t entrysize = pgstat_kind_info_for(kind)->pending_size; + + Assert(entrysize != (size_t) -1); + + entry_ref->pending = MemoryContextAllocZero(pgStatPendingContext, entrysize); + dlist_push_tail(&pgStatPending, &entry_ref->pending_node); + } + + return entry_ref; } /* - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the bgwriter statistics struct. + * Return an existing stats entry, or NULL. + * + * This should only be used for helper function for pgstatfuncs.c - outside of + * that it shouldn't be needed. */ -PgStat_BgWriterStats * -pgstat_fetch_stat_bgwriter(void) +PgStat_EntryRef * +pgstat_fetch_pending_entry(PgStat_Kind kind, Oid dboid, Oid objoid) { - backend_read_statsfile(); + PgStat_EntryRef *entry_ref; - return &globalStats.bgwriter; + entry_ref = pgstat_get_entry_ref(kind, dboid, objoid, false, NULL); + + if (entry_ref == NULL || entry_ref->pending == NULL) + return NULL; + + return entry_ref; +} + +void +pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) +{ + PgStat_Kind kind = entry_ref->shared_entry->key.kind; + const PgStat_KindInfo *kind_info = pgstat_kind_info_for(kind); + void *pending_data = entry_ref->pending; + + Assert(pending_data != NULL); + /* !fixed_amount stats should be handled explicitly */ + Assert(!pgstat_kind_info_for(kind)->fixed_amount); + + if (kind_info->delete_pending_cb) + kind_info->delete_pending_cb(entry_ref); + + pfree(pending_data); + entry_ref->pending = NULL; + + dlist_delete(&entry_ref->pending_node); } /* - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the checkpointer statistics struct. + * Flush out pending stats for database objects (databases, relations, + * functions). */ -PgStat_CheckpointerStats * -pgstat_fetch_stat_checkpointer(void) +static bool +pgstat_flush_pending_entries(bool nowait) { - backend_read_statsfile(); + bool have_pending = false; + dlist_node *cur = NULL; - return &globalStats.checkpointer; -} + /* + * Need to be a bit careful iterating over the list of pending entries. + * Processing a pending entry may queue further pending entries to the end + * of the list that we want to process, so a simple iteration won't do. + * Further complicating matters is that we want to delete the current + * entry in each iteration from the list if we flushed successfully. + * + * So we just keep track of the next pointer in each loop iteration. + */ + if (!dlist_is_empty(&pgStatPending)) + cur = dlist_head_node(&pgStatPending); -/* - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the WAL statistics struct. - */ -PgStat_WalStats * -pgstat_fetch_stat_wal(void) -{ - backend_read_statsfile(); + while (cur) + { + PgStat_EntryRef *entry_ref = + dlist_container(PgStat_EntryRef, pending_node, cur); + PgStat_HashKey key = entry_ref->shared_entry->key; + PgStat_Kind kind = key.kind; + const PgStat_KindInfo *kind_info = pgstat_kind_info_for(kind); + bool did_flush; + dlist_node *next; - return &walStats; -} + Assert(!kind_info->fixed_amount); + Assert(kind_info->flush_pending_cb != NULL); -/* - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the slru statistics struct. - */ -PgStat_SLRUStats * -pgstat_fetch_slru(void) -{ - backend_read_statsfile(); + /* flush the stats, if possible */ + did_flush = kind_info->flush_pending_cb(entry_ref, nowait); - return slruStats; -} + Assert(did_flush || nowait); -/* - * Support function for the SQL-callable pgstat* functions. Returns - * a pointer to the replication slot statistics struct. - */ -PgStat_StatReplSlotEntry * -pgstat_fetch_replslot(NameData slotname) -{ - backend_read_statsfile(); + /* determine next entry, before deleting the pending entry */ + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; - return pgstat_get_replslot_entry(slotname, false); -} + /* if successfully flushed, remove entry */ + if (did_flush) + pgstat_delete_pending_entry(entry_ref); + else + have_pending = true; -/* - * Support function for the SQL-callable pgstat* functions. Returns - * the collected statistics for one subscription or NULL. - */ -PgStat_StatSubEntry * -pgstat_fetch_stat_subscription(Oid subid) -{ - /* Load the stats file if needed */ - backend_read_statsfile(); + cur = next; + } - return pgstat_get_subscription_entry(subid, false); + Assert(dlist_is_empty(&pgStatPending) == !have_pending); + + return have_pending; } @@ -1390,16 +1106,56 @@ pgstat_fetch_stat_subscription(Oid subid) * ------------------------------------------------------------ */ -/* - * Create pgStatLocalContext, if not already done. - */ -static void -pgstat_setup_memcxt(void) +PgStat_Kind +pgstat_kind_from_str(char *kind_str) { - if (!pgStatLocalContext) - pgStatLocalContext = AllocSetContextCreate(TopMemoryContext, - "Statistics snapshot", - ALLOCSET_SMALL_SIZES); + for (int kind = PGSTAT_KIND_FIRST; kind <= PGSTAT_KIND_LAST; kind++) + { + if (pg_strcasecmp(kind_str, pgstat_kind_infos[kind].name) == 0) + return kind; + } + + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("invalid statistic kind: \"%s\"", kind_str))); + return PGSTAT_KIND_DATABASE; /* avoid compiler warnings */ +} + +static inline bool +pgstat_kind_valid(int ikind) +{ + return ikind >= PGSTAT_KIND_FIRST && ikind <= PGSTAT_KIND_LAST; +} + +const PgStat_KindInfo * +pgstat_kind_info_for(PgStat_Kind kind) +{ + AssertArg(pgstat_kind_valid(kind)); + + return &pgstat_kind_infos[kind]; +} + +/* + * The length of the data portion of a shared memory stats entry (i.e. without + * transient data such as refcoutns, lwlocks, ...). + */ +static inline size_t +shared_stat_entry_len(PgStat_Kind kind) +{ + return pgstat_kind_info_for(kind)->shared_data_len; +} + +/* + * Returns a pointer to the data portion of a shared memory stats entry. + */ +static inline void * +shared_stat_entry_data(PgStat_Kind kind, PgStatShared_Common *entry) +{ + size_t off = pgstat_kind_info_for(kind)->shared_data_off; + + Assert(off != 0 && off < PG_UINT32_MAX); + + return ((char *) (entry)) + off; } /* @@ -1415,642 +1171,44 @@ pgstat_assert_is_up(void) } #endif -/* - * Set common header fields in a statistics message - */ -void -pgstat_setheader(PgStat_MsgHdr *hdr, StatMsgType mtype) -{ - hdr->m_type = mtype; -} - - -/* - * Send out one statistics message to the collector - */ -void -pgstat_send(void *msg, int len) -{ - int rc; - - pgstat_assert_is_up(); - - if (pgStatSock == PGINVALID_SOCKET) - return; - - ((PgStat_MsgHdr *) msg)->m_size = len; - - /* We'll retry after EINTR, but ignore all other failures */ - do - { - rc = send(pgStatSock, msg, len, 0); - } while (rc < 0 && errno == EINTR); - -#ifdef USE_ASSERT_CHECKING - /* In debug builds, log send failures ... */ - if (rc < 0) - elog(LOG, "could not send to statistics collector: %m"); -#endif -} - -/* - * Start up the statistics collector process. This is the body of the - * postmaster child process. - * - * The argc/argv parameters are valid only in EXEC_BACKEND case. - */ -NON_EXEC_STATIC void -PgstatCollectorMain(int argc, char *argv[]) -{ - int len; - PgStat_Msg msg; - int wr; - WaitEvent event; - WaitEventSet *wes; - - /* - * Ignore all signals usually bound to some action in the postmaster, - * except SIGHUP and SIGQUIT. Note we don't need a SIGUSR1 handler to - * support latch operations, because we only use a local latch. - */ - pqsignal(SIGHUP, SignalHandlerForConfigReload); - pqsignal(SIGINT, SIG_IGN); - pqsignal(SIGTERM, SIG_IGN); - pqsignal(SIGQUIT, SignalHandlerForShutdownRequest); - pqsignal(SIGALRM, SIG_IGN); - pqsignal(SIGPIPE, SIG_IGN); - pqsignal(SIGUSR1, SIG_IGN); - pqsignal(SIGUSR2, SIG_IGN); - /* Reset some signals that are accepted by postmaster but not here */ - pqsignal(SIGCHLD, SIG_DFL); - PG_SETMASK(&UnBlockSig); - - MyBackendType = B_STATS_COLLECTOR; - init_ps_display(NULL); - - /* - * Read in existing stats files or initialize the stats to zero. - */ - pgStatRunningInCollector = true; - pgStatDBHash = pgstat_read_statsfiles(InvalidOid, true, true); - - /* Prepare to wait for our latch or data in our socket. */ - wes = CreateWaitEventSet(CurrentMemoryContext, 3); - AddWaitEventToSet(wes, WL_LATCH_SET, PGINVALID_SOCKET, MyLatch, NULL); - AddWaitEventToSet(wes, WL_POSTMASTER_DEATH, PGINVALID_SOCKET, NULL, NULL); - AddWaitEventToSet(wes, WL_SOCKET_READABLE, pgStatSock, NULL, NULL); - - /* - * Loop to process messages until we get SIGQUIT or detect ungraceful - * death of our parent postmaster. - * - * For performance reasons, we don't want to do ResetLatch/WaitLatch after - * every message; instead, do that only after a recv() fails to obtain a - * message. (This effectively means that if backends are sending us stuff - * like mad, we won't notice postmaster death until things slack off a - * bit; which seems fine.) To do that, we have an inner loop that - * iterates as long as recv() succeeds. We do check ConfigReloadPending - * inside the inner loop, which means that such interrupts will get - * serviced but the latch won't get cleared until next time there is a - * break in the action. - */ - for (;;) - { - /* Clear any already-pending wakeups */ - ResetLatch(MyLatch); - - /* - * Quit if we get SIGQUIT from the postmaster. - */ - if (ShutdownRequestPending) - break; - - /* - * Inner loop iterates as long as we keep getting messages, or until - * ShutdownRequestPending becomes set. - */ - while (!ShutdownRequestPending) - { - /* - * Reload configuration if we got SIGHUP from the postmaster. - */ - if (ConfigReloadPending) - { - ConfigReloadPending = false; - ProcessConfigFile(PGC_SIGHUP); - } - - /* - * Write the stats file(s) if a new request has arrived that is - * not satisfied by existing file(s). - */ - if (pgstat_write_statsfile_needed()) - pgstat_write_statsfiles(false, false); - - /* - * Try to receive and process a message. This will not block, - * since the socket is set to non-blocking mode. - * - * XXX On Windows, we have to force pgwin32_recv to cooperate, - * despite the previous use of pg_set_noblock() on the socket. - * This is extremely broken and should be fixed someday. - */ -#ifdef WIN32 - pgwin32_noblock = 1; -#endif - - len = recv(pgStatSock, (char *) &msg, - sizeof(PgStat_Msg), 0); - -#ifdef WIN32 - pgwin32_noblock = 0; -#endif - - if (len < 0) - { - if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) - break; /* out of inner loop */ - ereport(ERROR, - (errcode_for_socket_access(), - errmsg("could not read statistics message: %m"))); - } - - /* - * We ignore messages that are smaller than our common header - */ - if (len < sizeof(PgStat_MsgHdr)) - continue; - - /* - * The received length must match the length in the header - */ - if (msg.msg_hdr.m_size != len) - continue; - - /* - * O.K. - we accept this message. Process it. - */ - switch (msg.msg_hdr.m_type) - { - case PGSTAT_MTYPE_DUMMY: - break; - - case PGSTAT_MTYPE_INQUIRY: - pgstat_recv_inquiry(&msg.msg_inquiry, len); - break; - - case PGSTAT_MTYPE_TABSTAT: - pgstat_recv_tabstat(&msg.msg_tabstat, len); - break; - - case PGSTAT_MTYPE_TABPURGE: - pgstat_recv_tabpurge(&msg.msg_tabpurge, len); - break; - - case PGSTAT_MTYPE_DROPDB: - pgstat_recv_dropdb(&msg.msg_dropdb, len); - break; - - case PGSTAT_MTYPE_RESETCOUNTER: - pgstat_recv_resetcounter(&msg.msg_resetcounter, len); - break; - - case PGSTAT_MTYPE_RESETSHAREDCOUNTER: - pgstat_recv_resetsharedcounter(&msg.msg_resetsharedcounter, - len); - break; - - case PGSTAT_MTYPE_RESETSINGLECOUNTER: - pgstat_recv_resetsinglecounter(&msg.msg_resetsinglecounter, - len); - break; - - case PGSTAT_MTYPE_RESETSLRUCOUNTER: - pgstat_recv_resetslrucounter(&msg.msg_resetslrucounter, - len); - break; - - case PGSTAT_MTYPE_RESETREPLSLOTCOUNTER: - pgstat_recv_resetreplslotcounter(&msg.msg_resetreplslotcounter, - len); - break; - - case PGSTAT_MTYPE_RESETSUBCOUNTER: - pgstat_recv_resetsubcounter(&msg.msg_resetsubcounter, len); - break; - - case PGSTAT_MTYPE_AUTOVAC_START: - pgstat_recv_autovac(&msg.msg_autovacuum_start, len); - break; - - case PGSTAT_MTYPE_VACUUM: - pgstat_recv_vacuum(&msg.msg_vacuum, len); - break; - - case PGSTAT_MTYPE_ANALYZE: - pgstat_recv_analyze(&msg.msg_analyze, len); - break; - - case PGSTAT_MTYPE_ARCHIVER: - pgstat_recv_archiver(&msg.msg_archiver, len); - break; - - case PGSTAT_MTYPE_BGWRITER: - pgstat_recv_bgwriter(&msg.msg_bgwriter, len); - break; - - case PGSTAT_MTYPE_CHECKPOINTER: - pgstat_recv_checkpointer(&msg.msg_checkpointer, len); - break; - - case PGSTAT_MTYPE_WAL: - pgstat_recv_wal(&msg.msg_wal, len); - break; - - case PGSTAT_MTYPE_SLRU: - pgstat_recv_slru(&msg.msg_slru, len); - break; - - case PGSTAT_MTYPE_FUNCSTAT: - pgstat_recv_funcstat(&msg.msg_funcstat, len); - break; - - case PGSTAT_MTYPE_FUNCPURGE: - pgstat_recv_funcpurge(&msg.msg_funcpurge, len); - break; - - case PGSTAT_MTYPE_RECOVERYCONFLICT: - pgstat_recv_recoveryconflict(&msg.msg_recoveryconflict, - len); - break; - - case PGSTAT_MTYPE_DEADLOCK: - pgstat_recv_deadlock(&msg.msg_deadlock, len); - break; - - case PGSTAT_MTYPE_TEMPFILE: - pgstat_recv_tempfile(&msg.msg_tempfile, len); - break; - - case PGSTAT_MTYPE_CHECKSUMFAILURE: - pgstat_recv_checksum_failure(&msg.msg_checksumfailure, - len); - break; - - case PGSTAT_MTYPE_REPLSLOT: - pgstat_recv_replslot(&msg.msg_replslot, len); - break; - - case PGSTAT_MTYPE_CONNECT: - pgstat_recv_connect(&msg.msg_connect, len); - break; - - case PGSTAT_MTYPE_DISCONNECT: - pgstat_recv_disconnect(&msg.msg_disconnect, len); - break; - - case PGSTAT_MTYPE_SUBSCRIPTIONDROP: - pgstat_recv_subscription_drop(&msg.msg_subscriptiondrop, len); - break; - - case PGSTAT_MTYPE_SUBSCRIPTIONERROR: - pgstat_recv_subscription_error(&msg.msg_subscriptionerror, len); - break; - - default: - break; - } - } /* end of inner message-processing loop */ - - /* Sleep until there's something to do */ -#ifndef WIN32 - wr = WaitEventSetWait(wes, -1L, &event, 1, WAIT_EVENT_PGSTAT_MAIN); -#else - - /* - * Windows, at least in its Windows Server 2003 R2 incarnation, - * sometimes loses FD_READ events. Waking up and retrying the recv() - * fixes that, so don't sleep indefinitely. This is a crock of the - * first water, but until somebody wants to debug exactly what's - * happening there, this is the best we can do. The two-second - * timeout matches our pre-9.2 behavior, and needs to be short enough - * to not provoke "using stale statistics" complaints from - * backend_read_statsfile. - */ - wr = WaitEventSetWait(wes, 2 * 1000L /* msec */ , &event, 1, - WAIT_EVENT_PGSTAT_MAIN); -#endif - - /* - * Emergency bailout if postmaster has died. This is to avoid the - * necessity for manual cleanup of all postmaster children. - */ - if (wr == 1 && event.events == WL_POSTMASTER_DEATH) - break; - } /* end of outer loop */ - - /* - * Save the final stats to reuse at next startup. - */ - pgstat_write_statsfiles(true, true); - - FreeWaitEventSet(wes); - - exit(0); -} - -/* - * Subroutine to clear stats in a database entry - * - * Tables and functions hashes are initialized to empty. - */ -static void -reset_dbentry_counters(PgStat_StatDBEntry *dbentry) -{ - HASHCTL hash_ctl; - - dbentry->n_xact_commit = 0; - dbentry->n_xact_rollback = 0; - dbentry->n_blocks_fetched = 0; - dbentry->n_blocks_hit = 0; - dbentry->n_tuples_returned = 0; - dbentry->n_tuples_fetched = 0; - dbentry->n_tuples_inserted = 0; - dbentry->n_tuples_updated = 0; - dbentry->n_tuples_deleted = 0; - dbentry->last_autovac_time = 0; - dbentry->n_conflict_tablespace = 0; - dbentry->n_conflict_lock = 0; - dbentry->n_conflict_snapshot = 0; - dbentry->n_conflict_bufferpin = 0; - dbentry->n_conflict_startup_deadlock = 0; - dbentry->n_temp_files = 0; - dbentry->n_temp_bytes = 0; - dbentry->n_deadlocks = 0; - dbentry->n_checksum_failures = 0; - dbentry->last_checksum_failure = 0; - dbentry->n_block_read_time = 0; - dbentry->n_block_write_time = 0; - dbentry->n_sessions = 0; - dbentry->total_session_time = 0; - dbentry->total_active_time = 0; - dbentry->total_idle_in_xact_time = 0; - dbentry->n_sessions_abandoned = 0; - dbentry->n_sessions_fatal = 0; - dbentry->n_sessions_killed = 0; - - dbentry->stat_reset_timestamp = GetCurrentTimestamp(); - dbentry->stats_timestamp = 0; - - hash_ctl.keysize = sizeof(Oid); - hash_ctl.entrysize = sizeof(PgStat_StatTabEntry); - dbentry->tables = hash_create("Per-database table", - PGSTAT_TAB_HASH_SIZE, - &hash_ctl, - HASH_ELEM | HASH_BLOBS); - - hash_ctl.keysize = sizeof(Oid); - hash_ctl.entrysize = sizeof(PgStat_StatFuncEntry); - dbentry->functions = hash_create("Per-database function", - PGSTAT_FUNCTION_HASH_SIZE, - &hash_ctl, - HASH_ELEM | HASH_BLOBS); -} - -/* - * Lookup the hash table entry for the specified database. If no hash - * table entry exists, initialize it, if the create parameter is true. - * Else, return NULL. - */ -static PgStat_StatDBEntry * -pgstat_get_db_entry(Oid databaseid, bool create) -{ - PgStat_StatDBEntry *result; - bool found; - HASHACTION action = (create ? HASH_ENTER : HASH_FIND); - - /* Lookup or create the hash table entry for this database */ - result = (PgStat_StatDBEntry *) hash_search(pgStatDBHash, - &databaseid, - action, &found); - - if (!create && !found) - return NULL; - - /* - * If not found, initialize the new one. This creates empty hash tables - * for tables and functions, too. - */ - if (!found) - reset_dbentry_counters(result); - - return result; -} - -/* - * Lookup the hash table entry for the specified table. If no hash - * table entry exists, initialize it, if the create parameter is true. - * Else, return NULL. - */ -static PgStat_StatTabEntry * -pgstat_get_tab_entry(PgStat_StatDBEntry *dbentry, Oid tableoid, bool create) -{ - PgStat_StatTabEntry *result; - bool found; - HASHACTION action = (create ? HASH_ENTER : HASH_FIND); - - /* Lookup or create the hash table entry for this table */ - result = (PgStat_StatTabEntry *) hash_search(dbentry->tables, - &tableoid, - action, &found); - - if (!create && !found) - return NULL; - - /* If not found, initialize the new one. */ - if (!found) - { - result->numscans = 0; - result->tuples_returned = 0; - result->tuples_fetched = 0; - result->tuples_inserted = 0; - result->tuples_updated = 0; - result->tuples_deleted = 0; - result->tuples_hot_updated = 0; - result->n_live_tuples = 0; - result->n_dead_tuples = 0; - result->changes_since_analyze = 0; - result->inserts_since_vacuum = 0; - result->blocks_fetched = 0; - result->blocks_hit = 0; - result->vacuum_timestamp = 0; - result->vacuum_count = 0; - result->autovac_vacuum_timestamp = 0; - result->autovac_vacuum_count = 0; - result->analyze_timestamp = 0; - result->analyze_count = 0; - result->autovac_analyze_timestamp = 0; - result->autovac_analyze_count = 0; - } - - return result; -} - -/* - * Return the entry of replication slot stats with the given name. Return - * NULL if not found and the caller didn't request to create it. - * - * create tells whether to create the new slot entry if it is not found. - */ -static PgStat_StatReplSlotEntry * -pgstat_get_replslot_entry(NameData name, bool create) -{ - PgStat_StatReplSlotEntry *slotent; - bool found; - - if (replSlotStatHash == NULL) - { - HASHCTL hash_ctl; - - /* - * Quick return NULL if the hash table is empty and the caller didn't - * request to create the entry. - */ - if (!create) - return NULL; - - hash_ctl.keysize = sizeof(NameData); - hash_ctl.entrysize = sizeof(PgStat_StatReplSlotEntry); - replSlotStatHash = hash_create("Replication slots hash", - PGSTAT_REPLSLOT_HASH_SIZE, - &hash_ctl, - HASH_ELEM | HASH_BLOBS); - } - - slotent = (PgStat_StatReplSlotEntry *) hash_search(replSlotStatHash, - (void *) &name, - create ? HASH_ENTER : HASH_FIND, - &found); - - if (!slotent) - { - /* not found */ - Assert(!create && !found); - return NULL; - } - - /* initialize the entry */ - if (create && !found) - { - namestrcpy(&(slotent->slotname), NameStr(name)); - pgstat_reset_replslot(slotent, 0); - } - - return slotent; -} - -/* - * Reset the given replication slot stats. - */ -static void -pgstat_reset_replslot(PgStat_StatReplSlotEntry *slotent, TimestampTz ts) -{ - /* reset only counters. Don't clear slot name */ - slotent->spill_txns = 0; - slotent->spill_count = 0; - slotent->spill_bytes = 0; - slotent->stream_txns = 0; - slotent->stream_count = 0; - slotent->stream_bytes = 0; - slotent->total_txns = 0; - slotent->total_bytes = 0; - slotent->stat_reset_timestamp = ts; -} - -/* - * Return the subscription statistics entry with the given subscription OID. - * If no subscription entry exists, initialize it, if the create parameter is - * true. Else, return NULL. - */ -static PgStat_StatSubEntry * -pgstat_get_subscription_entry(Oid subid, bool create) -{ - PgStat_StatSubEntry *subentry; - bool found; - HASHACTION action = (create ? HASH_ENTER : HASH_FIND); - - if (subscriptionStatHash == NULL) - { - HASHCTL hash_ctl; - - /* - * Quick return NULL if the hash table is empty and the caller didn't - * request to create the entry. - */ - if (!create) - return NULL; - - hash_ctl.keysize = sizeof(Oid); - hash_ctl.entrysize = sizeof(PgStat_StatSubEntry); - subscriptionStatHash = hash_create("Subscription hash", - PGSTAT_SUBSCRIPTION_HASH_SIZE, - &hash_ctl, - HASH_ELEM | HASH_BLOBS); - } - - subentry = (PgStat_StatSubEntry *) hash_search(subscriptionStatHash, - (void *) &subid, - action, &found); - - if (!create && !found) - return NULL; - - /* If not found, initialize the new one */ - if (!found) - pgstat_reset_subscription(subentry, 0); - - return subentry; -} - -/* - * Reset the given subscription stats. - */ -static void -pgstat_reset_subscription(PgStat_StatSubEntry *subentry, TimestampTz ts) -{ - subentry->apply_error_count = 0; - subentry->sync_error_count = 0; - subentry->stat_reset_timestamp = ts; -} - /* ------------------------------------------------------------ * reading and writing of on-disk stats file * ------------------------------------------------------------ */ +/* helpers for pgstat_write_statsfile() */ +static void +write_chunk(FILE *fpout, void *ptr, size_t len) +{ + int rc; + + rc = fwrite(ptr, len, 1, fpout); + + /* we'll check for errors with ferror once at the end */ + (void) rc; +} + +#define write_chunk_s(fpout, ptr) write_chunk(fpout, ptr, sizeof(*ptr)) + /* - * Write the global statistics file, as well as requested DB files. - * - * 'permanent' specifies writing to the permanent files not temporary ones. - * When true (happens only when the collector is shutting down), also remove - * the temporary files so that backends starting up under a new postmaster - * can't read old data before the new collector is ready. - * - * When 'allDbs' is false, only the requested databases (listed in - * pending_write_requests) will be written; otherwise, all databases - * will be written. + * This function is called in the last process that is accessing the shared + * stats so locking is not required. */ static void -pgstat_write_statsfiles(bool permanent, bool allDbs) +pgstat_write_statsfile(void) { - HASH_SEQ_STATUS hstat; - PgStat_StatDBEntry *dbentry; FILE *fpout; int32 format_id; - const char *tmpfile = permanent ? PGSTAT_STAT_PERMANENT_TMPFILE : pgstat_stat_tmpname; - const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename; - int rc; + const char *tmpfile = PGSTAT_STAT_PERMANENT_TMPFILE; + const char *statfile = PGSTAT_STAT_PERMANENT_FILENAME; + dshash_seq_status hstat; + PgStatShared_HashEntry *ps; + + pgstat_assert_is_up(); + + /* we're shutting down, so it's ok to just override this */ + pgstat_fetch_consistency = PGSTAT_FETCH_CONSISTENCY_NONE; elog(DEBUG2, "writing stats file \"%s\"", statfile); @@ -2067,2077 +1225,350 @@ pgstat_write_statsfiles(bool permanent, bool allDbs) return; } - /* - * Set the timestamp of the stats file. - */ - globalStats.stats_timestamp = GetCurrentTimestamp(); - /* * Write the file header --- currently just a format ID. */ format_id = PGSTAT_FILE_FORMAT_ID; - rc = fwrite(&format_id, sizeof(format_id), 1, fpout); - (void) rc; /* we'll check for error with ferror */ + write_chunk_s(fpout, &format_id); /* - * Write global stats struct + * XXX: The following could now be generalized to just iterate over + * pgstat_kind_infos instead of knowing about the different kinds of + * stats. */ - rc = fwrite(&globalStats, sizeof(globalStats), 1, fpout); - (void) rc; /* we'll check for error with ferror */ /* * Write archiver stats struct */ - rc = fwrite(&archiverStats, sizeof(archiverStats), 1, fpout); - (void) rc; /* we'll check for error with ferror */ + pgstat_build_snapshot_global(PGSTAT_KIND_ARCHIVER); + write_chunk_s(fpout, &pgStatLocal.snapshot.archiver); /* - * Write WAL stats struct + * Write bgwriter stats struct */ - rc = fwrite(&walStats, sizeof(walStats), 1, fpout); - (void) rc; /* we'll check for error with ferror */ + pgstat_build_snapshot_global(PGSTAT_KIND_BGWRITER); + write_chunk_s(fpout, &pgStatLocal.snapshot.bgwriter); + + /* + * Write checkpointer stats struct + */ + pgstat_build_snapshot_global(PGSTAT_KIND_CHECKPOINTER); + write_chunk_s(fpout, &pgStatLocal.snapshot.checkpointer); /* * Write SLRU stats struct */ - rc = fwrite(slruStats, sizeof(slruStats), 1, fpout); - (void) rc; /* we'll check for error with ferror */ + pgstat_build_snapshot_global(PGSTAT_KIND_SLRU); + write_chunk_s(fpout, &pgStatLocal.snapshot.slru); /* - * Walk through the database table. + * Write WAL stats struct */ - hash_seq_init(&hstat, pgStatDBHash); - while ((dbentry = (PgStat_StatDBEntry *) hash_seq_search(&hstat)) != NULL) + pgstat_build_snapshot_global(PGSTAT_KIND_WAL); + write_chunk_s(fpout, &pgStatLocal.snapshot.wal); + + /* + * Walk through the stats entries + */ + dshash_seq_init(&hstat, pgStatLocal.shared_hash, false); + while ((ps = dshash_seq_next(&hstat)) != NULL) { - /* - * Write out the table and function stats for this DB into the - * appropriate per-DB stat file, if required. - */ - if (allDbs || pgstat_db_requested(dbentry->databaseid)) - { - /* Make DB's timestamp consistent with the global stats */ - dbentry->stats_timestamp = globalStats.stats_timestamp; - - pgstat_write_db_statsfile(dbentry, permanent); - } - - /* - * Write out the DB entry. We don't write the tables or functions - * pointers, since they're of no use to any other process. - */ - fputc('D', fpout); - rc = fwrite(dbentry, offsetof(PgStat_StatDBEntry, tables), 1, fpout); - (void) rc; /* we'll check for error with ferror */ - } - - /* - * Write replication slot stats struct - */ - if (replSlotStatHash) - { - PgStat_StatReplSlotEntry *slotent; - - hash_seq_init(&hstat, replSlotStatHash); - while ((slotent = (PgStat_StatReplSlotEntry *) hash_seq_search(&hstat)) != NULL) - { - fputc('R', fpout); - rc = fwrite(slotent, sizeof(PgStat_StatReplSlotEntry), 1, fpout); - (void) rc; /* we'll check for error with ferror */ - } - } - - /* - * Write subscription stats struct - */ - if (subscriptionStatHash) - { - PgStat_StatSubEntry *subentry; - - hash_seq_init(&hstat, subscriptionStatHash); - while ((subentry = (PgStat_StatSubEntry *) hash_seq_search(&hstat)) != NULL) - { - fputc('S', fpout); - rc = fwrite(subentry, sizeof(PgStat_StatSubEntry), 1, fpout); - (void) rc; /* we'll check for error with ferror */ - } - } - - /* - * No more output to be done. Close the temp file and replace the old - * pgstat.stat with it. The ferror() check replaces testing for error - * after each individual fputc or fwrite above. - */ - fputc('E', fpout); - - if (ferror(fpout)) - { - ereport(LOG, - (errcode_for_file_access(), - errmsg("could not write temporary statistics file \"%s\": %m", - tmpfile))); - FreeFile(fpout); - unlink(tmpfile); - } - else if (FreeFile(fpout) < 0) - { - ereport(LOG, - (errcode_for_file_access(), - errmsg("could not close temporary statistics file \"%s\": %m", - tmpfile))); - unlink(tmpfile); - } - else if (rename(tmpfile, statfile) < 0) - { - ereport(LOG, - (errcode_for_file_access(), - errmsg("could not rename temporary statistics file \"%s\" to \"%s\": %m", - tmpfile, statfile))); - unlink(tmpfile); - } - - if (permanent) - unlink(pgstat_stat_filename); - - /* - * Now throw away the list of requests. Note that requests sent after we - * started the write are still waiting on the network socket. - */ - list_free(pending_write_requests); - pending_write_requests = NIL; -} - -/* - * return the filename for a DB stat file; filename is the output buffer, - * of length len. - */ -static void -get_dbstat_filename(bool permanent, bool tempname, Oid databaseid, - char *filename, int len) -{ - int printed; - - /* NB -- pgstat_reset_remove_files knows about the pattern this uses */ - printed = snprintf(filename, len, "%s/db_%u.%s", - permanent ? PGSTAT_STAT_PERMANENT_DIRECTORY : - pgstat_stat_directory, - databaseid, - tempname ? "tmp" : "stat"); - if (printed >= len) - elog(ERROR, "overlength pgstat path"); -} - -/* - * Write the stat file for a single database. - * - * If writing to the permanent file (happens when the collector is - * shutting down only), remove the temporary file so that backends - * starting up under a new postmaster can't read the old data before - * the new collector is ready. - */ -static void -pgstat_write_db_statsfile(PgStat_StatDBEntry *dbentry, bool permanent) -{ - HASH_SEQ_STATUS tstat; - HASH_SEQ_STATUS fstat; - PgStat_StatTabEntry *tabentry; - PgStat_StatFuncEntry *funcentry; - FILE *fpout; - int32 format_id; - Oid dbid = dbentry->databaseid; - int rc; - char tmpfile[MAXPGPATH]; - char statfile[MAXPGPATH]; - - get_dbstat_filename(permanent, true, dbid, tmpfile, MAXPGPATH); - get_dbstat_filename(permanent, false, dbid, statfile, MAXPGPATH); - - elog(DEBUG2, "writing stats file \"%s\"", statfile); - - /* - * Open the statistics temp file to write out the current values. - */ - fpout = AllocateFile(tmpfile, PG_BINARY_W); - if (fpout == NULL) - { - ereport(LOG, - (errcode_for_file_access(), - errmsg("could not open temporary statistics file \"%s\": %m", - tmpfile))); - return; - } - - /* - * Write the file header --- currently just a format ID. - */ - format_id = PGSTAT_FILE_FORMAT_ID; - rc = fwrite(&format_id, sizeof(format_id), 1, fpout); - (void) rc; /* we'll check for error with ferror */ - - /* - * Walk through the database's access stats per table. - */ - hash_seq_init(&tstat, dbentry->tables); - while ((tabentry = (PgStat_StatTabEntry *) hash_seq_search(&tstat)) != NULL) - { - fputc('T', fpout); - rc = fwrite(tabentry, sizeof(PgStat_StatTabEntry), 1, fpout); - (void) rc; /* we'll check for error with ferror */ - } - - /* - * Walk through the database's function stats table. - */ - hash_seq_init(&fstat, dbentry->functions); - while ((funcentry = (PgStat_StatFuncEntry *) hash_seq_search(&fstat)) != NULL) - { - fputc('F', fpout); - rc = fwrite(funcentry, sizeof(PgStat_StatFuncEntry), 1, fpout); - (void) rc; /* we'll check for error with ferror */ - } - - /* - * No more output to be done. Close the temp file and replace the old - * pgstat.stat with it. The ferror() check replaces testing for error - * after each individual fputc or fwrite above. - */ - fputc('E', fpout); - - if (ferror(fpout)) - { - ereport(LOG, - (errcode_for_file_access(), - errmsg("could not write temporary statistics file \"%s\": %m", - tmpfile))); - FreeFile(fpout); - unlink(tmpfile); - } - else if (FreeFile(fpout) < 0) - { - ereport(LOG, - (errcode_for_file_access(), - errmsg("could not close temporary statistics file \"%s\": %m", - tmpfile))); - unlink(tmpfile); - } - else if (rename(tmpfile, statfile) < 0) - { - ereport(LOG, - (errcode_for_file_access(), - errmsg("could not rename temporary statistics file \"%s\" to \"%s\": %m", - tmpfile, statfile))); - unlink(tmpfile); - } - - if (permanent) - { - get_dbstat_filename(false, false, dbid, statfile, MAXPGPATH); - - elog(DEBUG2, "removing temporary stats file \"%s\"", statfile); - unlink(statfile); - } -} - -/* - * Reads in some existing statistics collector files and returns the - * databases hash table that is the top level of the data. - * - * If 'onlydb' is not InvalidOid, it means we only want data for that DB - * plus the shared catalogs ("DB 0"). We'll still populate the DB hash - * table for all databases, but we don't bother even creating table/function - * hash tables for other databases. - * - * 'permanent' specifies reading from the permanent files not temporary ones. - * When true (happens only when the collector is starting up), remove the - * files after reading; the in-memory status is now authoritative, and the - * files would be out of date in case somebody else reads them. - * - * If a 'deep' read is requested, table/function stats are read, otherwise - * the table/function hash tables remain empty. - */ -static HTAB * -pgstat_read_statsfiles(Oid onlydb, bool permanent, bool deep) -{ - PgStat_StatDBEntry *dbentry; - PgStat_StatDBEntry dbbuf; - HASHCTL hash_ctl; - HTAB *dbhash; - FILE *fpin; - int32 format_id; - bool found; - const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename; - int i; - TimestampTz ts; - - /* - * The tables will live in pgStatLocalContext. - */ - pgstat_setup_memcxt(); - - /* - * Create the DB hashtable - */ - hash_ctl.keysize = sizeof(Oid); - hash_ctl.entrysize = sizeof(PgStat_StatDBEntry); - hash_ctl.hcxt = pgStatLocalContext; - dbhash = hash_create("Databases hash", PGSTAT_DB_HASH_SIZE, &hash_ctl, - HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); - - /* - * Clear out global, archiver, WAL and SLRU statistics so they start from - * zero in case we can't load an existing statsfile. - */ - memset(&globalStats, 0, sizeof(globalStats)); - memset(&archiverStats, 0, sizeof(archiverStats)); - memset(&walStats, 0, sizeof(walStats)); - memset(&slruStats, 0, sizeof(slruStats)); - - /* - * Set the current timestamp (will be kept only in case we can't load an - * existing statsfile). - */ - ts = GetCurrentTimestamp(); - globalStats.bgwriter.stat_reset_timestamp = ts; - archiverStats.stat_reset_timestamp = ts; - walStats.stat_reset_timestamp = ts; - - /* - * Set the same reset timestamp for all SLRU items too. - */ - for (i = 0; i < SLRU_NUM_ELEMENTS; i++) - slruStats[i].stat_reset_timestamp = ts; - - /* - * Try to open the stats file. If it doesn't exist, the backends simply - * return zero for anything and the collector simply starts from scratch - * with empty counters. - * - * ENOENT is a possibility if the stats collector is not running or has - * not yet written the stats file the first time. Any other failure - * condition is suspicious. - */ - if ((fpin = AllocateFile(statfile, PG_BINARY_R)) == NULL) - { - if (errno != ENOENT) - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errcode_for_file_access(), - errmsg("could not open statistics file \"%s\": %m", - statfile))); - return dbhash; - } - - /* - * Verify it's of the expected format. - */ - if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) || - format_id != PGSTAT_FILE_FORMAT_ID) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", statfile))); - goto done; - } - - /* - * Read global stats struct - */ - if (fread(&globalStats, 1, sizeof(globalStats), fpin) != sizeof(globalStats)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", statfile))); - memset(&globalStats, 0, sizeof(globalStats)); - goto done; - } - - /* - * In the collector, disregard the timestamp we read from the permanent - * stats file; we should be willing to write a temp stats file immediately - * upon the first request from any backend. This only matters if the old - * file's timestamp is less than PGSTAT_STAT_INTERVAL ago, but that's not - * an unusual scenario. - */ - if (pgStatRunningInCollector) - globalStats.stats_timestamp = 0; - - /* - * Read archiver stats struct - */ - if (fread(&archiverStats, 1, sizeof(archiverStats), fpin) != sizeof(archiverStats)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", statfile))); - memset(&archiverStats, 0, sizeof(archiverStats)); - goto done; - } - - /* - * Read WAL stats struct - */ - if (fread(&walStats, 1, sizeof(walStats), fpin) != sizeof(walStats)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", statfile))); - memset(&walStats, 0, sizeof(walStats)); - goto done; - } - - /* - * Read SLRU stats struct - */ - if (fread(slruStats, 1, sizeof(slruStats), fpin) != sizeof(slruStats)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", statfile))); - memset(&slruStats, 0, sizeof(slruStats)); - goto done; - } - - /* - * We found an existing collector stats file. Read it and put all the - * hashtable entries into place. - */ - for (;;) - { - switch (fgetc(fpin)) - { - /* - * 'D' A PgStat_StatDBEntry struct describing a database - * follows. - */ - case 'D': - if (fread(&dbbuf, 1, offsetof(PgStat_StatDBEntry, tables), - fpin) != offsetof(PgStat_StatDBEntry, tables)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - goto done; - } - - /* - * Add to the DB hash - */ - dbentry = (PgStat_StatDBEntry *) hash_search(dbhash, - (void *) &dbbuf.databaseid, - HASH_ENTER, - &found); - if (found) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - goto done; - } - - memcpy(dbentry, &dbbuf, sizeof(PgStat_StatDBEntry)); - dbentry->tables = NULL; - dbentry->functions = NULL; - - /* - * In the collector, disregard the timestamp we read from the - * permanent stats file; we should be willing to write a temp - * stats file immediately upon the first request from any - * backend. - */ - if (pgStatRunningInCollector) - dbentry->stats_timestamp = 0; - - /* - * Don't create tables/functions hashtables for uninteresting - * databases. - */ - if (onlydb != InvalidOid) - { - if (dbbuf.databaseid != onlydb && - dbbuf.databaseid != InvalidOid) - break; - } - - hash_ctl.keysize = sizeof(Oid); - hash_ctl.entrysize = sizeof(PgStat_StatTabEntry); - hash_ctl.hcxt = pgStatLocalContext; - dbentry->tables = hash_create("Per-database table", - PGSTAT_TAB_HASH_SIZE, - &hash_ctl, - HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); - - hash_ctl.keysize = sizeof(Oid); - hash_ctl.entrysize = sizeof(PgStat_StatFuncEntry); - hash_ctl.hcxt = pgStatLocalContext; - dbentry->functions = hash_create("Per-database function", - PGSTAT_FUNCTION_HASH_SIZE, - &hash_ctl, - HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); - - /* - * If requested, read the data from the database-specific - * file. Otherwise we just leave the hashtables empty. - */ - if (deep) - pgstat_read_db_statsfile(dbentry->databaseid, - dbentry->tables, - dbentry->functions, - permanent); - - break; - - /* - * 'R' A PgStat_StatReplSlotEntry struct describing a - * replication slot follows. - */ - case 'R': - { - PgStat_StatReplSlotEntry slotbuf; - PgStat_StatReplSlotEntry *slotent; - - if (fread(&slotbuf, 1, sizeof(PgStat_StatReplSlotEntry), fpin) - != sizeof(PgStat_StatReplSlotEntry)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - goto done; - } - - /* Create hash table if we don't have it already. */ - if (replSlotStatHash == NULL) - { - HASHCTL hash_ctl; - - hash_ctl.keysize = sizeof(NameData); - hash_ctl.entrysize = sizeof(PgStat_StatReplSlotEntry); - hash_ctl.hcxt = pgStatLocalContext; - replSlotStatHash = hash_create("Replication slots hash", - PGSTAT_REPLSLOT_HASH_SIZE, - &hash_ctl, - HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); - } - - slotent = (PgStat_StatReplSlotEntry *) hash_search(replSlotStatHash, - (void *) &slotbuf.slotname, - HASH_ENTER, NULL); - memcpy(slotent, &slotbuf, sizeof(PgStat_StatReplSlotEntry)); - break; - } - - /* - * 'S' A PgStat_StatSubEntry struct describing subscription - * statistics. - */ - case 'S': - { - PgStat_StatSubEntry subbuf; - PgStat_StatSubEntry *subentry; - - if (fread(&subbuf, 1, sizeof(PgStat_StatSubEntry), fpin) - != sizeof(PgStat_StatSubEntry)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - goto done; - } - - if (subscriptionStatHash == NULL) - { - HASHCTL hash_ctl; - - hash_ctl.keysize = sizeof(Oid); - hash_ctl.entrysize = sizeof(PgStat_StatSubEntry); - hash_ctl.hcxt = pgStatLocalContext; - subscriptionStatHash = hash_create("Subscription hash", - PGSTAT_SUBSCRIPTION_HASH_SIZE, - &hash_ctl, - HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); - } - - subentry = (PgStat_StatSubEntry *) hash_search(subscriptionStatHash, - (void *) &subbuf.subid, - HASH_ENTER, NULL); - - memcpy(subentry, &subbuf, sizeof(subbuf)); - break; - } - - case 'E': - goto done; - - default: - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - goto done; - } - } - -done: - FreeFile(fpin); - - /* If requested to read the permanent file, also get rid of it. */ - if (permanent) - { - elog(DEBUG2, "removing permanent stats file \"%s\"", statfile); - unlink(statfile); - } - - return dbhash; -} - - -/* - * Reads in the existing statistics collector file for the given database, - * filling the passed-in tables and functions hash tables. - * - * As in pgstat_read_statsfiles, if the permanent file is requested, it is - * removed after reading. - * - * Note: this code has the ability to skip storing per-table or per-function - * data, if NULL is passed for the corresponding hashtable. That's not used - * at the moment though. - */ -static void -pgstat_read_db_statsfile(Oid databaseid, HTAB *tabhash, HTAB *funchash, - bool permanent) -{ - PgStat_StatTabEntry *tabentry; - PgStat_StatTabEntry tabbuf; - PgStat_StatFuncEntry funcbuf; - PgStat_StatFuncEntry *funcentry; - FILE *fpin; - int32 format_id; - bool found; - char statfile[MAXPGPATH]; - - get_dbstat_filename(permanent, false, databaseid, statfile, MAXPGPATH); - - /* - * Try to open the stats file. If it doesn't exist, the backends simply - * return zero for anything and the collector simply starts from scratch - * with empty counters. - * - * ENOENT is a possibility if the stats collector is not running or has - * not yet written the stats file the first time. Any other failure - * condition is suspicious. - */ - if ((fpin = AllocateFile(statfile, PG_BINARY_R)) == NULL) - { - if (errno != ENOENT) - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errcode_for_file_access(), - errmsg("could not open statistics file \"%s\": %m", - statfile))); - return; - } - - /* - * Verify it's of the expected format. - */ - if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) || - format_id != PGSTAT_FILE_FORMAT_ID) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", statfile))); - goto done; - } - - /* - * We found an existing collector stats file. Read it and put all the - * hashtable entries into place. - */ - for (;;) - { - switch (fgetc(fpin)) - { - /* - * 'T' A PgStat_StatTabEntry follows. - */ - case 'T': - if (fread(&tabbuf, 1, sizeof(PgStat_StatTabEntry), - fpin) != sizeof(PgStat_StatTabEntry)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - goto done; - } - - /* - * Skip if table data not wanted. - */ - if (tabhash == NULL) - break; - - tabentry = (PgStat_StatTabEntry *) hash_search(tabhash, - (void *) &tabbuf.tableid, - HASH_ENTER, &found); - - if (found) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - goto done; - } - - memcpy(tabentry, &tabbuf, sizeof(tabbuf)); - break; - - /* - * 'F' A PgStat_StatFuncEntry follows. - */ - case 'F': - if (fread(&funcbuf, 1, sizeof(PgStat_StatFuncEntry), - fpin) != sizeof(PgStat_StatFuncEntry)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - goto done; - } - - /* - * Skip if function data not wanted. - */ - if (funchash == NULL) - break; - - funcentry = (PgStat_StatFuncEntry *) hash_search(funchash, - (void *) &funcbuf.functionid, - HASH_ENTER, &found); - - if (found) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - goto done; - } - - memcpy(funcentry, &funcbuf, sizeof(funcbuf)); - break; - - /* - * 'E' The EOF marker of a complete stats file. - */ - case 'E': - goto done; - - default: - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - goto done; - } - } - -done: - FreeFile(fpin); - - if (permanent) - { - elog(DEBUG2, "removing permanent stats file \"%s\"", statfile); - unlink(statfile); - } -} - -/* - * Attempt to determine the timestamp of the last db statfile write. - * Returns true if successful; the timestamp is stored in *ts. The caller must - * rely on timestamp stored in *ts iff the function returns true. - * - * This needs to be careful about handling databases for which no stats file - * exists, such as databases without a stat entry or those not yet written: - * - * - if there's a database entry in the global file, return the corresponding - * stats_timestamp value. - * - * - if there's no db stat entry (e.g. for a new or inactive database), - * there's no stats_timestamp value, but also nothing to write so we return - * the timestamp of the global statfile. - */ -static bool -pgstat_read_db_statsfile_timestamp(Oid databaseid, bool permanent, - TimestampTz *ts) -{ - PgStat_StatDBEntry dbentry; - PgStat_GlobalStats myGlobalStats; - PgStat_ArchiverStats myArchiverStats; - PgStat_WalStats myWalStats; - PgStat_SLRUStats mySLRUStats[SLRU_NUM_ELEMENTS]; - PgStat_StatReplSlotEntry myReplSlotStats; - PgStat_StatSubEntry mySubStats; - FILE *fpin; - int32 format_id; - const char *statfile = permanent ? PGSTAT_STAT_PERMANENT_FILENAME : pgstat_stat_filename; - - /* - * Try to open the stats file. As above, anything but ENOENT is worthy of - * complaining about. - */ - if ((fpin = AllocateFile(statfile, PG_BINARY_R)) == NULL) - { - if (errno != ENOENT) - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errcode_for_file_access(), - errmsg("could not open statistics file \"%s\": %m", - statfile))); - return false; - } - - /* - * Verify it's of the expected format. - */ - if (fread(&format_id, 1, sizeof(format_id), fpin) != sizeof(format_id) || - format_id != PGSTAT_FILE_FORMAT_ID) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", statfile))); - FreeFile(fpin); - return false; - } - - /* - * Read global stats struct - */ - if (fread(&myGlobalStats, 1, sizeof(myGlobalStats), - fpin) != sizeof(myGlobalStats)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", statfile))); - FreeFile(fpin); - return false; - } - - /* - * Read archiver stats struct - */ - if (fread(&myArchiverStats, 1, sizeof(myArchiverStats), - fpin) != sizeof(myArchiverStats)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", statfile))); - FreeFile(fpin); - return false; - } - - /* - * Read WAL stats struct - */ - if (fread(&myWalStats, 1, sizeof(myWalStats), fpin) != sizeof(myWalStats)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", statfile))); - FreeFile(fpin); - return false; - } - - /* - * Read SLRU stats struct - */ - if (fread(mySLRUStats, 1, sizeof(mySLRUStats), fpin) != sizeof(mySLRUStats)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", statfile))); - FreeFile(fpin); - return false; - } - - /* By default, we're going to return the timestamp of the global file. */ - *ts = myGlobalStats.stats_timestamp; - - /* - * We found an existing collector stats file. Read it and look for a - * record for the requested database. If found, use its timestamp. - */ - for (;;) - { - switch (fgetc(fpin)) - { - /* - * 'D' A PgStat_StatDBEntry struct describing a database - * follows. - */ - case 'D': - if (fread(&dbentry, 1, offsetof(PgStat_StatDBEntry, tables), - fpin) != offsetof(PgStat_StatDBEntry, tables)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - FreeFile(fpin); - return false; - } - - /* - * If this is the DB we're looking for, save its timestamp and - * we're done. - */ - if (dbentry.databaseid == databaseid) - { - *ts = dbentry.stats_timestamp; - goto done; - } - - break; - - /* - * 'R' A PgStat_StatReplSlotEntry struct describing a - * replication slot follows. - */ - case 'R': - if (fread(&myReplSlotStats, 1, sizeof(PgStat_StatReplSlotEntry), fpin) - != sizeof(PgStat_StatReplSlotEntry)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - FreeFile(fpin); - return false; - } - break; - - /* - * 'S' A PgStat_StatSubEntry struct describing subscription - * statistics follows. - */ - case 'S': - if (fread(&mySubStats, 1, sizeof(PgStat_StatSubEntry), fpin) - != sizeof(PgStat_StatSubEntry)) - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - FreeFile(fpin); - return false; - } - break; - - case 'E': - goto done; - - default: - { - ereport(pgStatRunningInCollector ? LOG : WARNING, - (errmsg("corrupted statistics file \"%s\"", - statfile))); - FreeFile(fpin); - return false; - } - } - } - -done: - FreeFile(fpin); - return true; -} - -/* - * If not already done, read the statistics collector stats file into - * some hash tables. The results will be kept until pgstat_clear_snapshot() - * is called (typically, at end of transaction). - */ -static void -backend_read_statsfile(void) -{ - TimestampTz min_ts = 0; - TimestampTz ref_ts = 0; - Oid inquiry_db; - int count; - - pgstat_assert_is_up(); - - /* already read it? */ - if (pgStatDBHash) - return; - Assert(!pgStatRunningInCollector); - - /* - * In a normal backend, we check staleness of the data for our own DB, and - * so we send MyDatabaseId in inquiry messages. In the autovac launcher, - * check staleness of the shared-catalog data, and send InvalidOid in - * inquiry messages so as not to force writing unnecessary data. - */ - if (IsAutoVacuumLauncherProcess()) - inquiry_db = InvalidOid; - else - inquiry_db = MyDatabaseId; - - /* - * Loop until fresh enough stats file is available or we ran out of time. - * The stats inquiry message is sent repeatedly in case collector drops - * it; but not every single time, as that just swamps the collector. - */ - for (count = 0; count < PGSTAT_POLL_LOOP_COUNT; count++) - { - bool ok; - TimestampTz file_ts = 0; - TimestampTz cur_ts; + PgStatShared_Common *shstats; + const PgStat_KindInfo *kind_info = NULL; CHECK_FOR_INTERRUPTS(); - ok = pgstat_read_db_statsfile_timestamp(inquiry_db, false, &file_ts); + /* we may have some "dropped" entries not yet removed, skip them */ + Assert(!ps->dropped); + if (ps->dropped) + continue; - cur_ts = GetCurrentTimestamp(); - /* Calculate min acceptable timestamp, if we didn't already */ - if (count == 0 || cur_ts < ref_ts) + shstats = (PgStatShared_Common *) dsa_get_address(pgStatLocal.dsa, ps->body); + + kind_info = pgstat_kind_info_for(ps->key.kind); + + /* if not dropped the valid-entry refcount should exist */ + Assert(pg_atomic_read_u32(&ps->refcount) > 0); + + if (!kind_info->to_serialized_name) { - /* - * We set the minimum acceptable timestamp to PGSTAT_STAT_INTERVAL - * msec before now. This indirectly ensures that the collector - * needn't write the file more often than PGSTAT_STAT_INTERVAL. In - * an autovacuum worker, however, we want a lower delay to avoid - * using stale data, so we use PGSTAT_RETRY_DELAY (since the - * number of workers is low, this shouldn't be a problem). - * - * We don't recompute min_ts after sleeping, except in the - * unlikely case that cur_ts went backwards. So we might end up - * accepting a file a bit older than PGSTAT_STAT_INTERVAL. In - * practice that shouldn't happen, though, as long as the sleep - * time is less than PGSTAT_STAT_INTERVAL; and we don't want to - * tell the collector that our cutoff time is less than what we'd - * actually accept. - */ - ref_ts = cur_ts; - if (IsAutoVacuumWorkerProcess()) - min_ts = TimestampTzPlusMilliseconds(ref_ts, - -PGSTAT_RETRY_DELAY); - else - min_ts = TimestampTzPlusMilliseconds(ref_ts, - -PGSTAT_STAT_INTERVAL); + /* normal stats entry, identified by PgStat_HashKey */ + fputc('S', fpout); + write_chunk_s(fpout, &ps->key); + } + else + { + /* stats entry identified by name on disk (e.g. slots) */ + NameData name; + + kind_info->to_serialized_name(shstats, &name); + + fputc('N', fpout); + write_chunk_s(fpout, &ps->key.kind); + write_chunk_s(fpout, &name); } - /* - * If the file timestamp is actually newer than cur_ts, we must have - * had a clock glitch (system time went backwards) or there is clock - * skew between our processor and the stats collector's processor. - * Accept the file, but send an inquiry message anyway to make - * pgstat_recv_inquiry do a sanity check on the collector's time. - */ - if (ok && file_ts > cur_ts) - { - /* - * A small amount of clock skew between processors isn't terribly - * surprising, but a large difference is worth logging. We - * arbitrarily define "large" as 1000 msec. - */ - if (file_ts >= TimestampTzPlusMilliseconds(cur_ts, 1000)) - { - char *filetime; - char *mytime; - - /* Copy because timestamptz_to_str returns a static buffer */ - filetime = pstrdup(timestamptz_to_str(file_ts)); - mytime = pstrdup(timestamptz_to_str(cur_ts)); - ereport(LOG, - (errmsg("statistics collector's time %s is later than backend local time %s", - filetime, mytime))); - pfree(filetime); - pfree(mytime); - } - - pgstat_send_inquiry(cur_ts, min_ts, inquiry_db); - break; - } - - /* Normal acceptance case: file is not older than cutoff time */ - if (ok && file_ts >= min_ts) - break; - - /* Not there or too old, so kick the collector and wait a bit */ - if ((count % PGSTAT_INQ_LOOP_COUNT) == 0) - pgstat_send_inquiry(cur_ts, min_ts, inquiry_db); - - pg_usleep(PGSTAT_RETRY_DELAY * 1000L); + /* Write except the header part of the entry */ + write_chunk(fpout, + shared_stat_entry_data(ps->key.kind, shstats), + shared_stat_entry_len(ps->key.kind)); } + dshash_seq_term(&hstat); - if (count >= PGSTAT_POLL_LOOP_COUNT) + /* + * No more output to be done. Close the temp file and replace the old + * pgstat.stat with it. The ferror() check replaces testing for error + * after each individual fputc or fwrite (in write_chunk()) above. + */ + fputc('E', fpout); + + if (ferror(fpout)) + { ereport(LOG, - (errmsg("using stale statistics instead of current ones " - "because stats collector is not responding"))); - - /* - * Autovacuum launcher wants stats about all databases, but a shallow read - * is sufficient. Regular backends want a deep read for just the tables - * they can see (MyDatabaseId + shared catalogs). - */ - if (IsAutoVacuumLauncherProcess()) - pgStatDBHash = pgstat_read_statsfiles(InvalidOid, false, false); - else - pgStatDBHash = pgstat_read_statsfiles(MyDatabaseId, false, true); + (errcode_for_file_access(), + errmsg("could not write temporary statistics file \"%s\": %m", + tmpfile))); + FreeFile(fpout); + unlink(tmpfile); + } + else if (FreeFile(fpout) < 0) + { + ereport(LOG, + (errcode_for_file_access(), + errmsg("could not close temporary statistics file \"%s\": %m", + tmpfile))); + unlink(tmpfile); + } + else if (rename(tmpfile, statfile) < 0) + { + ereport(LOG, + (errcode_for_file_access(), + errmsg("could not rename temporary statistics file \"%s\" to \"%s\": %m", + tmpfile, statfile))); + unlink(tmpfile); + } } -/* - * Do we need to write out any stats files? - */ +/* helpers for pgstat_read_statsfile() */ static bool -pgstat_write_statsfile_needed(void) +read_chunk(FILE *fpin, void *ptr, size_t len) { - if (pending_write_requests != NIL) - return true; - - /* Everything was written recently */ - return false; + return fread(ptr, 1, len, fpin) == len; } -/* - * Checks whether stats for a particular DB need to be written to a file. - */ -static bool -pgstat_db_requested(Oid databaseid) -{ - /* - * If any requests are outstanding at all, we should write the stats for - * shared catalogs (the "database" with OID 0). This ensures that - * backends will see up-to-date stats for shared catalogs, even though - * they send inquiry messages mentioning only their own DB. - */ - if (databaseid == InvalidOid && pending_write_requests != NIL) - return true; - - /* Search to see if there's an open request to write this database. */ - if (list_member_oid(pending_write_requests, databaseid)) - return true; - - return false; -} - - -/* ------------------------------------------------------------ - * stats collector message processing functions - * ------------------------------------------------------------ - */ +#define read_chunk_s(fpin, ptr) read_chunk(fpin, ptr, sizeof(*ptr)) /* - * Process stat inquiry requests. + * Reads in existing statistics file into the shared stats hash. + * + * This function is called in the only process that is accessing the shared + * stats so locking is not required. */ static void -pgstat_recv_inquiry(PgStat_MsgInquiry *msg, int len) +pgstat_read_statsfile(void) { - PgStat_StatDBEntry *dbentry; - - elog(DEBUG2, "received inquiry for database %u", msg->databaseid); - - /* - * If there's already a write request for this DB, there's nothing to do. - * - * Note that if a request is found, we return early and skip the below - * check for clock skew. This is okay, since the only way for a DB - * request to be present in the list is that we have been here since the - * last write round. It seems sufficient to check for clock skew once per - * write round. - */ - if (list_member_oid(pending_write_requests, msg->databaseid)) - return; - - /* - * Check to see if we last wrote this database at a time >= the requested - * cutoff time. If so, this is a stale request that was generated before - * we updated the DB file, and we don't need to do so again. - * - * If the requestor's local clock time is older than stats_timestamp, we - * should suspect a clock glitch, ie system time going backwards; though - * the more likely explanation is just delayed message receipt. It is - * worth expending a GetCurrentTimestamp call to be sure, since a large - * retreat in the system clock reading could otherwise cause us to neglect - * to update the stats file for a long time. - */ - dbentry = pgstat_get_db_entry(msg->databaseid, false); - if (dbentry == NULL) - { - /* - * We have no data for this DB. Enter a write request anyway so that - * the global stats will get updated. This is needed to prevent - * backend_read_statsfile from waiting for data that we cannot supply, - * in the case of a new DB that nobody has yet reported any stats for. - * See the behavior of pgstat_read_db_statsfile_timestamp. - */ - } - else if (msg->clock_time < dbentry->stats_timestamp) - { - TimestampTz cur_ts = GetCurrentTimestamp(); - - if (cur_ts < dbentry->stats_timestamp) - { - /* - * Sure enough, time went backwards. Force a new stats file write - * to get back in sync; but first, log a complaint. - */ - char *writetime; - char *mytime; - - /* Copy because timestamptz_to_str returns a static buffer */ - writetime = pstrdup(timestamptz_to_str(dbentry->stats_timestamp)); - mytime = pstrdup(timestamptz_to_str(cur_ts)); - ereport(LOG, - (errmsg("stats_timestamp %s is later than collector's time %s for database %u", - writetime, mytime, dbentry->databaseid))); - pfree(writetime); - pfree(mytime); - } - else - { - /* - * Nope, it's just an old request. Assuming msg's clock_time is - * >= its cutoff_time, it must be stale, so we can ignore it. - */ - return; - } - } - else if (msg->cutoff_time <= dbentry->stats_timestamp) - { - /* Stale request, ignore it */ - return; - } - - /* - * We need to write this DB, so create a request. - */ - pending_write_requests = lappend_oid(pending_write_requests, - msg->databaseid); -} - -/* - * Count what the backend has done. - */ -static void -pgstat_recv_tabstat(PgStat_MsgTabstat *msg, int len) -{ - PgStat_StatDBEntry *dbentry; - PgStat_StatTabEntry *tabentry; - int i; + FILE *fpin; + int32 format_id; bool found; - - dbentry = pgstat_get_db_entry(msg->m_databaseid, true); - - /* - * Update database-wide stats. - */ - dbentry->n_xact_commit += (PgStat_Counter) (msg->m_xact_commit); - dbentry->n_xact_rollback += (PgStat_Counter) (msg->m_xact_rollback); - dbentry->n_block_read_time += msg->m_block_read_time; - dbentry->n_block_write_time += msg->m_block_write_time; - - dbentry->total_session_time += msg->m_session_time; - dbentry->total_active_time += msg->m_active_time; - dbentry->total_idle_in_xact_time += msg->m_idle_in_xact_time; - - /* - * Process all table entries in the message. - */ - for (i = 0; i < msg->m_nentries; i++) - { - PgStat_TableEntry *tabmsg = &(msg->m_entry[i]); - - tabentry = (PgStat_StatTabEntry *) hash_search(dbentry->tables, - (void *) &(tabmsg->t_id), - HASH_ENTER, &found); - - if (!found) - { - /* - * If it's a new table entry, initialize counters to the values we - * just got. - */ - tabentry->numscans = tabmsg->t_counts.t_numscans; - tabentry->tuples_returned = tabmsg->t_counts.t_tuples_returned; - tabentry->tuples_fetched = tabmsg->t_counts.t_tuples_fetched; - tabentry->tuples_inserted = tabmsg->t_counts.t_tuples_inserted; - tabentry->tuples_updated = tabmsg->t_counts.t_tuples_updated; - tabentry->tuples_deleted = tabmsg->t_counts.t_tuples_deleted; - tabentry->tuples_hot_updated = tabmsg->t_counts.t_tuples_hot_updated; - tabentry->n_live_tuples = tabmsg->t_counts.t_delta_live_tuples; - tabentry->n_dead_tuples = tabmsg->t_counts.t_delta_dead_tuples; - tabentry->changes_since_analyze = tabmsg->t_counts.t_changed_tuples; - tabentry->inserts_since_vacuum = tabmsg->t_counts.t_tuples_inserted; - tabentry->blocks_fetched = tabmsg->t_counts.t_blocks_fetched; - tabentry->blocks_hit = tabmsg->t_counts.t_blocks_hit; - - tabentry->vacuum_timestamp = 0; - tabentry->vacuum_count = 0; - tabentry->autovac_vacuum_timestamp = 0; - tabentry->autovac_vacuum_count = 0; - tabentry->analyze_timestamp = 0; - tabentry->analyze_count = 0; - tabentry->autovac_analyze_timestamp = 0; - tabentry->autovac_analyze_count = 0; - } - else - { - /* - * Otherwise add the values to the existing entry. - */ - tabentry->numscans += tabmsg->t_counts.t_numscans; - tabentry->tuples_returned += tabmsg->t_counts.t_tuples_returned; - tabentry->tuples_fetched += tabmsg->t_counts.t_tuples_fetched; - tabentry->tuples_inserted += tabmsg->t_counts.t_tuples_inserted; - tabentry->tuples_updated += tabmsg->t_counts.t_tuples_updated; - tabentry->tuples_deleted += tabmsg->t_counts.t_tuples_deleted; - tabentry->tuples_hot_updated += tabmsg->t_counts.t_tuples_hot_updated; - - /* - * If table was truncated/dropped, first reset the live/dead - * counters. - */ - if (tabmsg->t_counts.t_truncdropped) - { - tabentry->n_live_tuples = 0; - tabentry->n_dead_tuples = 0; - tabentry->inserts_since_vacuum = 0; - } - tabentry->n_live_tuples += tabmsg->t_counts.t_delta_live_tuples; - tabentry->n_dead_tuples += tabmsg->t_counts.t_delta_dead_tuples; - tabentry->changes_since_analyze += tabmsg->t_counts.t_changed_tuples; - tabentry->inserts_since_vacuum += tabmsg->t_counts.t_tuples_inserted; - tabentry->blocks_fetched += tabmsg->t_counts.t_blocks_fetched; - tabentry->blocks_hit += tabmsg->t_counts.t_blocks_hit; - } - - /* Clamp n_live_tuples in case of negative delta_live_tuples */ - tabentry->n_live_tuples = Max(tabentry->n_live_tuples, 0); - /* Likewise for n_dead_tuples */ - tabentry->n_dead_tuples = Max(tabentry->n_dead_tuples, 0); - - /* - * Add per-table stats to the per-database entry, too. - */ - dbentry->n_tuples_returned += tabmsg->t_counts.t_tuples_returned; - dbentry->n_tuples_fetched += tabmsg->t_counts.t_tuples_fetched; - dbentry->n_tuples_inserted += tabmsg->t_counts.t_tuples_inserted; - dbentry->n_tuples_updated += tabmsg->t_counts.t_tuples_updated; - dbentry->n_tuples_deleted += tabmsg->t_counts.t_tuples_deleted; - dbentry->n_blocks_fetched += tabmsg->t_counts.t_blocks_fetched; - dbentry->n_blocks_hit += tabmsg->t_counts.t_blocks_hit; - } -} - -/* - * Arrange for dead table removal. - */ -static void -pgstat_recv_tabpurge(PgStat_MsgTabpurge *msg, int len) -{ - PgStat_StatDBEntry *dbentry; - int i; - - dbentry = pgstat_get_db_entry(msg->m_databaseid, false); - - /* - * No need to purge if we don't even know the database. - */ - if (!dbentry || !dbentry->tables) - return; - - /* - * Process all table entries in the message. - */ - for (i = 0; i < msg->m_nentries; i++) - { - /* Remove from hashtable if present; we don't care if it's not. */ - (void) hash_search(dbentry->tables, - (void *) &(msg->m_tableid[i]), - HASH_REMOVE, NULL); - } -} - -/* - * Arrange for dead database removal - */ -static void -pgstat_recv_dropdb(PgStat_MsgDropdb *msg, int len) -{ - Oid dbid = msg->m_databaseid; - PgStat_StatDBEntry *dbentry; - - /* - * Lookup the database in the hashtable. - */ - dbentry = pgstat_get_db_entry(dbid, false); - - /* - * If found, remove it (along with the db statfile). - */ - if (dbentry) - { - char statfile[MAXPGPATH]; - - get_dbstat_filename(false, false, dbid, statfile, MAXPGPATH); - - elog(DEBUG2, "removing stats file \"%s\"", statfile); - unlink(statfile); - - if (dbentry->tables != NULL) - hash_destroy(dbentry->tables); - if (dbentry->functions != NULL) - hash_destroy(dbentry->functions); - - if (hash_search(pgStatDBHash, - (void *) &dbid, - HASH_REMOVE, NULL) == NULL) - ereport(ERROR, - (errmsg("database hash table corrupted during cleanup --- abort"))); - } -} - -/* - * Reset the statistics for the specified database. - */ -static void -pgstat_recv_resetcounter(PgStat_MsgResetcounter *msg, int len) -{ - PgStat_StatDBEntry *dbentry; - - /* - * Lookup the database in the hashtable. Nothing to do if not there. - */ - dbentry = pgstat_get_db_entry(msg->m_databaseid, false); - - if (!dbentry) - return; - - /* - * We simply throw away all the database's table entries by recreating a - * new hash table for them. - */ - if (dbentry->tables != NULL) - hash_destroy(dbentry->tables); - if (dbentry->functions != NULL) - hash_destroy(dbentry->functions); - - dbentry->tables = NULL; - dbentry->functions = NULL; - - /* - * Reset database-level stats, too. This creates empty hash tables for - * tables and functions. - */ - reset_dbentry_counters(dbentry); -} - -/* - * Reset some shared statistics of the cluster. - */ -static void -pgstat_recv_resetsharedcounter(PgStat_MsgResetsharedcounter *msg, int len) -{ - if (msg->m_resettarget == PGSTAT_KIND_BGWRITER || - msg->m_resettarget == PGSTAT_KIND_CHECKPOINTER) - { - /* - * Reset the global, bgwriter and checkpointer statistics for the - * cluster. - */ - memset(&globalStats, 0, sizeof(globalStats)); - globalStats.bgwriter.stat_reset_timestamp = GetCurrentTimestamp(); - } - else if (msg->m_resettarget == PGSTAT_KIND_ARCHIVER) - { - /* Reset the archiver statistics for the cluster. */ - memset(&archiverStats, 0, sizeof(archiverStats)); - archiverStats.stat_reset_timestamp = GetCurrentTimestamp(); - } - else if (msg->m_resettarget == PGSTAT_KIND_SLRU) - { - int i; - TimestampTz ts = GetCurrentTimestamp(); - - for (i = 0; i < SLRU_NUM_ELEMENTS; i++) - { - memset(&slruStats[i], 0, sizeof(slruStats[i])); - slruStats[i].stat_reset_timestamp = ts; - } - } - else if (msg->m_resettarget == PGSTAT_KIND_WAL) - { - /* Reset the WAL statistics for the cluster. */ - memset(&walStats, 0, sizeof(walStats)); - walStats.stat_reset_timestamp = GetCurrentTimestamp(); - } - - /* - * Presumably the sender of this message validated the target, don't - * complain here if it's not valid - */ -} - -/* - * Reset a statistics for a single object, which may be of current - * database or shared across all databases in the cluster. - */ -static void -pgstat_recv_resetsinglecounter(PgStat_MsgResetsinglecounter *msg, int len) -{ - PgStat_StatDBEntry *dbentry; - - if (IsSharedRelation(msg->m_objectid)) - dbentry = pgstat_get_db_entry(InvalidOid, false); - else - dbentry = pgstat_get_db_entry(msg->m_databaseid, false); - - if (!dbentry) - return; - - /* Set the reset timestamp for the whole database */ - dbentry->stat_reset_timestamp = GetCurrentTimestamp(); - - /* Remove object if it exists, ignore it if not */ - if (msg->m_resettype == PGSTAT_KIND_RELATION) - (void) hash_search(dbentry->tables, (void *) &(msg->m_objectid), - HASH_REMOVE, NULL); - else if (msg->m_resettype == PGSTAT_KIND_FUNCTION) - (void) hash_search(dbentry->functions, (void *) &(msg->m_objectid), - HASH_REMOVE, NULL); -} - -/* - * Reset some SLRU statistics of the cluster. - */ -static void -pgstat_recv_resetslrucounter(PgStat_MsgResetslrucounter *msg, int len) -{ - int i; + const char *statfile = PGSTAT_STAT_PERMANENT_FILENAME; + PgStat_ShmemControl *shmem = pgStatLocal.shmem; TimestampTz ts = GetCurrentTimestamp(); - for (i = 0; i < SLRU_NUM_ELEMENTS; i++) + /* shouldn't be called from postmaster */ + Assert(IsUnderPostmaster || !IsPostmasterEnvironment); + + elog(DEBUG2, "reading stats file \"%s\"", statfile); + + /* + * Try to open the stats file. If it doesn't exist, the backends simply + * returns zero for anything and statistics simply starts from scratch + * with empty counters. + * + * ENOENT is a possibility if stats collection was previously disabled or + * has not yet written the stats file for the first time. Any other + * failure condition is suspicious. + */ + if ((fpin = AllocateFile(statfile, PG_BINARY_R)) == NULL) { - /* reset entry with the given index */ - if (msg->m_index == i) - { - memset(&slruStats[i], 0, sizeof(slruStats[i])); - slruStats[i].stat_reset_timestamp = ts; - } - } -} - -/* - * Reset some replication slot statistics of the cluster. - */ -static void -pgstat_recv_resetreplslotcounter(PgStat_MsgResetreplslotcounter *msg, - int len) -{ - PgStat_StatReplSlotEntry *slotent; - TimestampTz ts; - - /* Return if we don't have replication slot statistics */ - if (replSlotStatHash == NULL) + if (errno != ENOENT) + ereport(LOG, + (errcode_for_file_access(), + errmsg("could not open statistics file \"%s\": %m", + statfile))); + pgstat_reset_all_stats(ts); return; - - ts = GetCurrentTimestamp(); - if (msg->clearall) - { - HASH_SEQ_STATUS sstat; - - hash_seq_init(&sstat, replSlotStatHash); - while ((slotent = (PgStat_StatReplSlotEntry *) hash_seq_search(&sstat)) != NULL) - pgstat_reset_replslot(slotent, ts); } - else - { - /* Get the slot statistics to reset */ - slotent = pgstat_get_replslot_entry(msg->m_slotname, false); - - /* - * Nothing to do if the given slot entry is not found. This could - * happen when the slot with the given name is removed and the - * corresponding statistics entry is also removed before receiving the - * reset message. - */ - if (!slotent) - return; - - /* Reset the stats for the requested replication slot */ - pgstat_reset_replslot(slotent, ts); - } -} - -/* - * Reset some subscription statistics of the cluster. - */ -static void -pgstat_recv_resetsubcounter(PgStat_MsgResetsubcounter *msg, int len) -{ - PgStat_StatSubEntry *subentry; - TimestampTz ts; - - /* Return if we don't have replication subscription statistics */ - if (subscriptionStatHash == NULL) - return; - - ts = GetCurrentTimestamp(); - if (!OidIsValid(msg->m_subid)) - { - HASH_SEQ_STATUS sstat; - - /* Clear all subscription counters */ - hash_seq_init(&sstat, subscriptionStatHash); - while ((subentry = (PgStat_StatSubEntry *) hash_seq_search(&sstat)) != NULL) - pgstat_reset_subscription(subentry, ts); - } - else - { - /* Get the subscription statistics to reset */ - subentry = pgstat_get_subscription_entry(msg->m_subid, false); - - /* - * Nothing to do if the given subscription entry is not found. This - * could happen when the subscription with the subid is removed and - * the corresponding statistics entry is also removed before receiving - * the reset message. - */ - if (!subentry) - return; - - /* Reset the stats for the requested subscription */ - pgstat_reset_subscription(subentry, ts); - } -} - -/* - * Process an autovacuum signaling message. - */ -static void -pgstat_recv_autovac(PgStat_MsgAutovacStart *msg, int len) -{ - PgStat_StatDBEntry *dbentry; /* - * Store the last autovacuum time in the database's hashtable entry. + * Verify it's of the expected format. */ - dbentry = pgstat_get_db_entry(msg->m_databaseid, true); - - dbentry->last_autovac_time = msg->m_start_time; -} - -/* - * Process a VACUUM message. - */ -static void -pgstat_recv_vacuum(PgStat_MsgVacuum *msg, int len) -{ - PgStat_StatDBEntry *dbentry; - PgStat_StatTabEntry *tabentry; + if (!read_chunk_s(fpin, &format_id) || + format_id != PGSTAT_FILE_FORMAT_ID) + goto error; /* - * Store the data in the table's hashtable entry. + * XXX: The following could now be generalized to just iterate over + * pgstat_kind_infos instead of knowing about the different kinds of + * stats. */ - dbentry = pgstat_get_db_entry(msg->m_databaseid, true); - - tabentry = pgstat_get_tab_entry(dbentry, msg->m_tableoid, true); - - tabentry->n_live_tuples = msg->m_live_tuples; - tabentry->n_dead_tuples = msg->m_dead_tuples; /* - * It is quite possible that a non-aggressive VACUUM ended up skipping - * various pages, however, we'll zero the insert counter here regardless. - * It's currently used only to track when we need to perform an "insert" - * autovacuum, which are mainly intended to freeze newly inserted tuples. - * Zeroing this may just mean we'll not try to vacuum the table again - * until enough tuples have been inserted to trigger another insert - * autovacuum. An anti-wraparound autovacuum will catch any persistent - * stragglers. + * Read archiver stats struct */ - tabentry->inserts_since_vacuum = 0; - - if (msg->m_autovacuum) - { - tabentry->autovac_vacuum_timestamp = msg->m_vacuumtime; - tabentry->autovac_vacuum_count++; - } - else - { - tabentry->vacuum_timestamp = msg->m_vacuumtime; - tabentry->vacuum_count++; - } -} - -/* - * Process an ANALYZE message. - */ -static void -pgstat_recv_analyze(PgStat_MsgAnalyze *msg, int len) -{ - PgStat_StatDBEntry *dbentry; - PgStat_StatTabEntry *tabentry; + if (!read_chunk_s(fpin, &shmem->archiver.stats)) + goto error; /* - * Store the data in the table's hashtable entry. + * Read bgwriter stats struct */ - dbentry = pgstat_get_db_entry(msg->m_databaseid, true); - - tabentry = pgstat_get_tab_entry(dbentry, msg->m_tableoid, true); - - tabentry->n_live_tuples = msg->m_live_tuples; - tabentry->n_dead_tuples = msg->m_dead_tuples; + if (!read_chunk_s(fpin, &shmem->bgwriter.stats)) + goto error; /* - * If commanded, reset changes_since_analyze to zero. This forgets any - * changes that were committed while the ANALYZE was in progress, but we - * have no good way to estimate how many of those there were. + * Read checkpointer stats struct */ - if (msg->m_resetcounter) - tabentry->changes_since_analyze = 0; + if (!read_chunk_s(fpin, &shmem->checkpointer.stats)) + goto error; - if (msg->m_autovacuum) + /* + * Read SLRU stats struct + */ + if (!read_chunk_s(fpin, &shmem->slru.stats)) + goto error; + + /* + * Read WAL stats struct + */ + if (!read_chunk_s(fpin, &shmem->wal.stats)) + goto error; + + /* + * We found an existing statistics file. Read it and put all the hash + * table entries into place. + */ + for (;;) { - tabentry->autovac_analyze_timestamp = msg->m_analyzetime; - tabentry->autovac_analyze_count++; - } - else - { - tabentry->analyze_timestamp = msg->m_analyzetime; - tabentry->analyze_count++; - } -} + char t = fgetc(fpin); -/* - * Process a ARCHIVER message. - */ -static void -pgstat_recv_archiver(PgStat_MsgArchiver *msg, int len) -{ - if (msg->m_failed) - { - /* Failed archival attempt */ - ++archiverStats.failed_count; - memcpy(archiverStats.last_failed_wal, msg->m_xlog, - sizeof(archiverStats.last_failed_wal)); - archiverStats.last_failed_timestamp = msg->m_timestamp; - } - else - { - /* Successful archival operation */ - ++archiverStats.archived_count; - memcpy(archiverStats.last_archived_wal, msg->m_xlog, - sizeof(archiverStats.last_archived_wal)); - archiverStats.last_archived_timestamp = msg->m_timestamp; - } -} - -/* - * Process a BGWRITER message. - */ -static void -pgstat_recv_bgwriter(PgStat_MsgBgWriter *msg, int len) -{ - globalStats.bgwriter.buf_written_clean += msg->m_buf_written_clean; - globalStats.bgwriter.maxwritten_clean += msg->m_maxwritten_clean; - globalStats.bgwriter.buf_alloc += msg->m_buf_alloc; -} - -/* - * Process a CHECKPOINTER message. - */ -static void -pgstat_recv_checkpointer(PgStat_MsgCheckpointer *msg, int len) -{ - globalStats.checkpointer.timed_checkpoints += msg->m_timed_checkpoints; - globalStats.checkpointer.requested_checkpoints += msg->m_requested_checkpoints; - globalStats.checkpointer.checkpoint_write_time += msg->m_checkpoint_write_time; - globalStats.checkpointer.checkpoint_sync_time += msg->m_checkpoint_sync_time; - globalStats.checkpointer.buf_written_checkpoints += msg->m_buf_written_checkpoints; - globalStats.checkpointer.buf_written_backend += msg->m_buf_written_backend; - globalStats.checkpointer.buf_fsync_backend += msg->m_buf_fsync_backend; -} - -/* - * Process a WAL message. - */ -static void -pgstat_recv_wal(PgStat_MsgWal *msg, int len) -{ - walStats.wal_records += msg->m_wal_records; - walStats.wal_fpi += msg->m_wal_fpi; - walStats.wal_bytes += msg->m_wal_bytes; - walStats.wal_buffers_full += msg->m_wal_buffers_full; - walStats.wal_write += msg->m_wal_write; - walStats.wal_sync += msg->m_wal_sync; - walStats.wal_write_time += msg->m_wal_write_time; - walStats.wal_sync_time += msg->m_wal_sync_time; -} - -/* - * Process a SLRU message. - */ -static void -pgstat_recv_slru(PgStat_MsgSLRU *msg, int len) -{ - slruStats[msg->m_index].blocks_zeroed += msg->m_blocks_zeroed; - slruStats[msg->m_index].blocks_hit += msg->m_blocks_hit; - slruStats[msg->m_index].blocks_read += msg->m_blocks_read; - slruStats[msg->m_index].blocks_written += msg->m_blocks_written; - slruStats[msg->m_index].blocks_exists += msg->m_blocks_exists; - slruStats[msg->m_index].flush += msg->m_flush; - slruStats[msg->m_index].truncate += msg->m_truncate; -} - -/* - * Process a RECOVERYCONFLICT message. - */ -static void -pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int len) -{ - PgStat_StatDBEntry *dbentry; - - dbentry = pgstat_get_db_entry(msg->m_databaseid, true); - - switch (msg->m_reason) - { - case PROCSIG_RECOVERY_CONFLICT_DATABASE: - - /* - * Since we drop the information about the database as soon as it - * replicates, there is no point in counting these conflicts. - */ - break; - case PROCSIG_RECOVERY_CONFLICT_TABLESPACE: - dbentry->n_conflict_tablespace++; - break; - case PROCSIG_RECOVERY_CONFLICT_LOCK: - dbentry->n_conflict_lock++; - break; - case PROCSIG_RECOVERY_CONFLICT_SNAPSHOT: - dbentry->n_conflict_snapshot++; - break; - case PROCSIG_RECOVERY_CONFLICT_BUFFERPIN: - dbentry->n_conflict_bufferpin++; - break; - case PROCSIG_RECOVERY_CONFLICT_STARTUP_DEADLOCK: - dbentry->n_conflict_startup_deadlock++; - break; - } -} - -/* - * Process a DEADLOCK message. - */ -static void -pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len) -{ - PgStat_StatDBEntry *dbentry; - - dbentry = pgstat_get_db_entry(msg->m_databaseid, true); - - dbentry->n_deadlocks++; -} - -/* - * Process a CHECKSUMFAILURE message. - */ -static void -pgstat_recv_checksum_failure(PgStat_MsgChecksumFailure *msg, int len) -{ - PgStat_StatDBEntry *dbentry; - - dbentry = pgstat_get_db_entry(msg->m_databaseid, true); - - dbentry->n_checksum_failures += msg->m_failurecount; - dbentry->last_checksum_failure = msg->m_failure_time; -} - -/* - * Process a REPLSLOT message. - */ -static void -pgstat_recv_replslot(PgStat_MsgReplSlot *msg, int len) -{ - if (msg->m_drop) - { - Assert(!msg->m_create); - - /* Remove the replication slot statistics with the given name */ - if (replSlotStatHash != NULL) - (void) hash_search(replSlotStatHash, - (void *) &(msg->m_slotname), - HASH_REMOVE, - NULL); - } - else - { - PgStat_StatReplSlotEntry *slotent; - - slotent = pgstat_get_replslot_entry(msg->m_slotname, true); - Assert(slotent); - - if (msg->m_create) + switch (t) { - /* - * If the message for dropping the slot with the same name gets - * lost, slotent has stats for the old slot. So we initialize all - * counters at slot creation. - */ - pgstat_reset_replslot(slotent, 0); - } - else - { - /* Update the replication slot statistics */ - slotent->spill_txns += msg->m_spill_txns; - slotent->spill_count += msg->m_spill_count; - slotent->spill_bytes += msg->m_spill_bytes; - slotent->stream_txns += msg->m_stream_txns; - slotent->stream_count += msg->m_stream_count; - slotent->stream_bytes += msg->m_stream_bytes; - slotent->total_txns += msg->m_total_txns; - slotent->total_bytes += msg->m_total_bytes; + case 'S': + case 'N': + { + PgStat_HashKey key; + PgStatShared_HashEntry *p; + PgStatShared_Common *header; + + CHECK_FOR_INTERRUPTS(); + + if (t == 'S') + { + /* normal stats entry, identified by PgStat_HashKey */ + if (!read_chunk_s(fpin, &key)) + goto error; + + if (!pgstat_kind_valid(key.kind)) + goto error; + } + else + { + /* stats entry identified by name on disk (e.g. slots) */ + const PgStat_KindInfo *kind_info = NULL; + PgStat_Kind kind; + NameData name; + + if (!read_chunk_s(fpin, &kind)) + goto error; + if (!read_chunk_s(fpin, &name)) + goto error; + if (!pgstat_kind_valid(kind)) + goto error; + + kind_info = pgstat_kind_info_for(kind); + + if (!kind_info->from_serialized_name) + goto error; + + if (!kind_info->from_serialized_name(&name, &key)) + { + /* skip over data for entry we don't care about */ + if (fseek(fpin, shared_stat_entry_len(kind), SEEK_CUR) != 0) + { + elog(LOG, "couldn't seek"); + goto error; + } + + continue; + } + + Assert(key.kind == kind); + } + + /* + * This intentionally doesn't use pgstat_get_entry_ref() - + * putting all stats into checkpointer's + * pgStatEntryRefHash would be wasted effort and memory. + */ + p = dshash_find_or_insert(pgStatLocal.shared_hash, &key, &found); + + /* don't allow duplicate entries */ + if (found) + { + dshash_release_lock(pgStatLocal.shared_hash, p); + elog(WARNING, "found duplicate stats entry %d/%u/%u", + key.kind, key.dboid, key.objoid); + goto error; + } + + header = pgstat_init_entry(key.kind, p); + dshash_release_lock(pgStatLocal.shared_hash, p); + + if (!read_chunk(fpin, + shared_stat_entry_data(key.kind, header), + shared_stat_entry_len(key.kind))) + goto error; + + break; + } + case 'E': + goto done; + + default: + goto error; } } + +done: + FreeFile(fpin); + + elog(DEBUG2, "removing permanent stats file \"%s\"", statfile); + unlink(statfile); + + return; + +error: + ereport(LOG, + (errmsg("corrupted statistics file \"%s\"", statfile))); + + /* Set the current timestamp as reset timestamp */ + pgstat_reset_all_stats(ts); + + goto done; } /* - * Process a CONNECT message. + * Helper to initially reset all stats */ static void -pgstat_recv_connect(PgStat_MsgConnect *msg, int len) +pgstat_reset_all_stats(TimestampTz ts) { - PgStat_StatDBEntry *dbentry; - - dbentry = pgstat_get_db_entry(msg->m_databaseid, true); - dbentry->n_sessions++; -} - -/* - * Process a DISCONNECT message. - */ -static void -pgstat_recv_disconnect(PgStat_MsgDisconnect *msg, int len) -{ - PgStat_StatDBEntry *dbentry; - - dbentry = pgstat_get_db_entry(msg->m_databaseid, true); - - switch (msg->m_cause) + for (int kind = PGSTAT_KIND_FIRST; kind <= PGSTAT_KIND_LAST; kind++) { - case DISCONNECT_NOT_YET: - case DISCONNECT_NORMAL: - /* we don't collect these */ - break; - case DISCONNECT_CLIENT_EOF: - dbentry->n_sessions_abandoned++; - break; - case DISCONNECT_FATAL: - dbentry->n_sessions_fatal++; - break; - case DISCONNECT_KILLED: - dbentry->n_sessions_killed++; - break; + const PgStat_KindInfo *kind_info = pgstat_kind_info_for(kind); + + if (!kind_info->fixed_amount) + continue; + + kind_info->reset_all_cb(ts); } } - -/* - * Process a TEMPFILE message. - */ -static void -pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len) -{ - PgStat_StatDBEntry *dbentry; - - dbentry = pgstat_get_db_entry(msg->m_databaseid, true); - - dbentry->n_temp_bytes += msg->m_filesize; - dbentry->n_temp_files += 1; -} - -/* - * Count what the backend has done. - */ -static void -pgstat_recv_funcstat(PgStat_MsgFuncstat *msg, int len) -{ - PgStat_FunctionEntry *funcmsg = &(msg->m_entry[0]); - PgStat_StatDBEntry *dbentry; - PgStat_StatFuncEntry *funcentry; - int i; - bool found; - - dbentry = pgstat_get_db_entry(msg->m_databaseid, true); - - /* - * Process all function entries in the message. - */ - for (i = 0; i < msg->m_nentries; i++, funcmsg++) - { - funcentry = (PgStat_StatFuncEntry *) hash_search(dbentry->functions, - (void *) &(funcmsg->f_id), - HASH_ENTER, &found); - - if (!found) - { - /* - * If it's a new function entry, initialize counters to the values - * we just got. - */ - funcentry->f_numcalls = funcmsg->f_numcalls; - funcentry->f_total_time = funcmsg->f_total_time; - funcentry->f_self_time = funcmsg->f_self_time; - } - else - { - /* - * Otherwise add the values to the existing entry. - */ - funcentry->f_numcalls += funcmsg->f_numcalls; - funcentry->f_total_time += funcmsg->f_total_time; - funcentry->f_self_time += funcmsg->f_self_time; - } - } -} - -/* - * Arrange for dead function removal. - */ -static void -pgstat_recv_funcpurge(PgStat_MsgFuncpurge *msg, int len) -{ - PgStat_StatDBEntry *dbentry; - int i; - - dbentry = pgstat_get_db_entry(msg->m_databaseid, false); - - /* - * No need to purge if we don't even know the database. - */ - if (!dbentry || !dbentry->functions) - return; - - /* - * Process all function entries in the message. - */ - for (i = 0; i < msg->m_nentries; i++) - { - /* Remove from hashtable if present; we don't care if it's not. */ - (void) hash_search(dbentry->functions, - (void *) &(msg->m_functionid[i]), - HASH_REMOVE, NULL); - } -} - -/* - * Process a SUBSCRIPTIONDROP message. - */ -static void -pgstat_recv_subscription_drop(PgStat_MsgSubscriptionDrop *msg, int len) -{ - /* Return if we don't have replication subscription statistics */ - if (subscriptionStatHash == NULL) - return; - - /* Remove from hashtable if present; we don't care if it's not */ - (void) hash_search(subscriptionStatHash, (void *) &(msg->m_subid), - HASH_REMOVE, NULL); -} - -/* - * Process a SUBSCRIPTIONERROR message. - */ -static void -pgstat_recv_subscription_error(PgStat_MsgSubscriptionError *msg, int len) -{ - PgStat_StatSubEntry *subentry; - - /* Get the subscription stats */ - subentry = pgstat_get_subscription_entry(msg->m_subid, true); - Assert(subentry); - - if (msg->m_is_apply_error) - subentry->apply_error_count++; - else - subentry->sync_error_count++; -} diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index 80bb2695998..4dc1a5f1db3 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -255,7 +255,6 @@ static pid_t StartupPID = 0, WalReceiverPID = 0, AutoVacPID = 0, PgArchPID = 0, - PgStatPID = 0, SysLoggerPID = 0; /* Startup process's status */ @@ -517,7 +516,6 @@ typedef struct PGPROC *AuxiliaryProcs; PGPROC *PreparedXactProcs; PMSignalData *PMSignalState; - InheritableSocket pgStatSock; pid_t PostmasterPid; TimestampTz PgStartTime; TimestampTz PgReloadTime; @@ -652,9 +650,8 @@ PostmasterMain(int argc, char *argv[]) * CAUTION: when changing this list, check for side-effects on the signal * handling setup of child processes. See tcop/postgres.c, * bootstrap/bootstrap.c, postmaster/bgwriter.c, postmaster/walwriter.c, - * postmaster/autovacuum.c, postmaster/pgarch.c, postmaster/pgstat.c, - * postmaster/syslogger.c, postmaster/bgworker.c and - * postmaster/checkpointer.c. + * postmaster/autovacuum.c, postmaster/pgarch.c, postmaster/syslogger.c, + * postmaster/bgworker.c and postmaster/checkpointer.c. */ pqinitmask(); PG_SETMASK(&BlockSig); @@ -1391,12 +1388,6 @@ PostmasterMain(int argc, char *argv[]) */ RemovePgTempFiles(); - /* - * Initialize stats collection subsystem (this does NOT start the - * collector process!) - */ - pgstat_init(); - /* * Initialize the autovacuum subsystem (again, no process start yet) */ @@ -1852,11 +1843,6 @@ ServerLoop(void) start_autovac_launcher = false; /* signal processed */ } - /* If we have lost the stats collector, try to start a new one */ - if (PgStatPID == 0 && - (pmState == PM_RUN || pmState == PM_HOT_STANDBY)) - PgStatPID = pgstat_start(); - /* If we have lost the archiver, try to start a new one. */ if (PgArchPID == 0 && PgArchStartupAllowed()) PgArchPID = StartArchiver(); @@ -2791,8 +2777,6 @@ SIGHUP_handler(SIGNAL_ARGS) signal_child(PgArchPID, SIGHUP); if (SysLoggerPID != 0) signal_child(SysLoggerPID, SIGHUP); - if (PgStatPID != 0) - signal_child(PgStatPID, SIGHUP); /* Reload authentication config files too */ if (!load_hba()) @@ -3121,8 +3105,6 @@ reaper(SIGNAL_ARGS) AutoVacPID = StartAutoVacLauncher(); if (PgArchStartupAllowed() && PgArchPID == 0) PgArchPID = StartArchiver(); - if (PgStatPID == 0) - PgStatPID = pgstat_start(); /* workers may be scheduled to start now */ maybe_start_bgworkers(); @@ -3189,13 +3171,6 @@ reaper(SIGNAL_ARGS) SignalChildren(SIGUSR2); pmState = PM_SHUTDOWN_2; - - /* - * We can also shut down the stats collector now; there's - * nothing left for it to do. - */ - if (PgStatPID != 0) - signal_child(PgStatPID, SIGQUIT); } else { @@ -3274,22 +3249,6 @@ reaper(SIGNAL_ARGS) continue; } - /* - * Was it the statistics collector? If so, just try to start a new - * one; no need to force reset of the rest of the system. (If fail, - * we'll try again in future cycles of the main loop.) - */ - if (pid == PgStatPID) - { - PgStatPID = 0; - if (!EXIT_STATUS_0(exitstatus)) - LogChildExit(LOG, _("statistics collector process"), - pid, exitstatus); - if (pmState == PM_RUN || pmState == PM_HOT_STANDBY) - PgStatPID = pgstat_start(); - continue; - } - /* Was it the system logger? If so, try to start a new one */ if (pid == SysLoggerPID) { @@ -3731,22 +3690,6 @@ HandleChildCrash(int pid, int exitstatus, const char *procname) signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT)); } - /* - * Force a power-cycle of the pgstat process too. (This isn't absolutely - * necessary, but it seems like a good idea for robustness, and it - * simplifies the state-machine logic in the case where a shutdown request - * arrives during crash processing.) - */ - if (PgStatPID != 0 && take_action) - { - ereport(DEBUG2, - (errmsg_internal("sending %s to process %d", - "SIGQUIT", - (int) PgStatPID))); - signal_child(PgStatPID, SIGQUIT); - allow_immediate_pgstat_restart(); - } - /* We do NOT restart the syslogger */ if (Shutdown != ImmediateShutdown) @@ -3968,12 +3911,10 @@ PostmasterStateMachine(void) FatalError = true; pmState = PM_WAIT_DEAD_END; - /* Kill the walsenders, archiver and stats collector too */ + /* Kill the walsenders and archiver too */ SignalChildren(SIGQUIT); if (PgArchPID != 0) signal_child(PgArchPID, SIGQUIT); - if (PgStatPID != 0) - signal_child(PgStatPID, SIGQUIT); } } } @@ -3997,8 +3938,7 @@ PostmasterStateMachine(void) { /* * PM_WAIT_DEAD_END state ends when the BackendList is entirely empty - * (ie, no dead_end children remain), and the archiver and stats - * collector are gone too. + * (ie, no dead_end children remain), and the archiver is gone too. * * The reason we wait for those two is to protect them against a new * postmaster starting conflicting subprocesses; this isn't an @@ -4008,8 +3948,7 @@ PostmasterStateMachine(void) * normal state transition leading up to PM_WAIT_DEAD_END, or during * FatalError processing. */ - if (dlist_is_empty(&BackendList) && - PgArchPID == 0 && PgStatPID == 0) + if (dlist_is_empty(&BackendList) && PgArchPID == 0) { /* These other guys should be dead already */ Assert(StartupPID == 0); @@ -4229,8 +4168,6 @@ TerminateChildren(int signal) signal_child(AutoVacPID, signal); if (PgArchPID != 0) signal_child(PgArchPID, signal); - if (PgStatPID != 0) - signal_child(PgStatPID, signal); } /* @@ -5162,12 +5099,6 @@ SubPostmasterMain(int argc, char *argv[]) StartBackgroundWorker(); } - if (strcmp(argv[1], "--forkcol") == 0) - { - /* Do not want to attach to shared memory */ - - PgstatCollectorMain(argc, argv); /* does not return */ - } if (strcmp(argv[1], "--forklog") == 0) { /* Do not want to attach to shared memory */ @@ -5271,12 +5202,6 @@ sigusr1_handler(SIGNAL_ARGS) if (CheckPostmasterSignal(PMSIGNAL_BEGIN_HOT_STANDBY) && pmState == PM_RECOVERY && Shutdown == NoShutdown) { - /* - * Likewise, start other special children as needed. - */ - Assert(PgStatPID == 0); - PgStatPID = pgstat_start(); - ereport(LOG, (errmsg("database system is ready to accept read-only connections"))); @@ -6192,7 +6117,6 @@ extern slock_t *ShmemLock; extern slock_t *ProcStructLock; extern PGPROC *AuxiliaryProcs; extern PMSignalData *PMSignalState; -extern pgsocket pgStatSock; extern pg_time_t first_syslogger_file_time; #ifndef WIN32 @@ -6248,8 +6172,6 @@ save_backend_variables(BackendParameters *param, Port *port, param->AuxiliaryProcs = AuxiliaryProcs; param->PreparedXactProcs = PreparedXactProcs; param->PMSignalState = PMSignalState; - if (!write_inheritable_socket(¶m->pgStatSock, pgStatSock, childPid)) - return false; param->PostmasterPid = PostmasterPid; param->PgStartTime = PgStartTime; @@ -6483,7 +6405,6 @@ restore_backend_variables(BackendParameters *param, Port *port) AuxiliaryProcs = param->AuxiliaryProcs; PreparedXactProcs = param->PreparedXactProcs; PMSignalState = param->PMSignalState; - read_inheritable_socket(&pgStatSock, ¶m->pgStatSock); PostmasterPid = param->PostmasterPid; PgStartTime = param->PgStartTime; @@ -6522,8 +6443,6 @@ restore_backend_variables(BackendParameters *param, Port *port) if (postmaster_alive_fds[1] >= 0) ReserveExternalFD(); #endif - if (pgStatSock != PGINVALID_SOCKET) - ReserveExternalFD(); } diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index 656ec8f5555..30e33dace33 100644 --- a/src/backend/replication/logical/logical.c +++ b/src/backend/replication/logical/logical.c @@ -1911,7 +1911,6 @@ UpdateDecodingStats(LogicalDecodingContext *ctx) (long long) rb->totalTxns, (long long) rb->totalBytes); - namestrcpy(&repSlotStat.slotname, NameStr(ctx->slot->data.name)); repSlotStat.spill_txns = rb->spillTxns; repSlotStat.spill_count = rb->spillCount; repSlotStat.spill_bytes = rb->spillBytes; diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index 5d3b3fe6db6..9979173953e 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -499,6 +499,14 @@ retry: /* We made this slot active, so it's ours now. */ MyReplicationSlot = s; + + /* + * The call to pgstat_acquire_replslot() protects against stats for + * a different slot, from before a restart or such, being present during + * pgstat_report_replslot(). + */ + if (SlotIsLogical(s)) + pgstat_acquire_replslot(s); } /* @@ -743,23 +751,13 @@ ReplicationSlotDropPtr(ReplicationSlot *slot) elog(DEBUG3, "replication slot drop: %s: removed directory", NameStr(slot->data.name)); /* - * Send a message to drop the replication slot to the stats collector. - * Since there is no guarantee of the order of message transfer on a UDP - * connection, it's possible that a message for creating a new slot - * reaches before a message for removing the old slot. We send the drop - * and create messages while holding ReplicationSlotAllocationLock to - * reduce that possibility. If the messages reached in reverse, we would - * lose one statistics update message. But the next update message will - * create the statistics for the replication slot. - * - * XXX In case, the messages for creation and drop slot of the same name - * get lost and create happens before (auto)vacuum cleans up the dead - * slot, the stats will be accumulated into the old slot. One can imagine - * having OIDs for each slot to avoid the accumulation of stats but that - * doesn't seem worth doing as in practice this won't happen frequently. + * Drop the statistics entry for the replication slot. Do this while + * holding ReplicationSlotAllocationLock so that we don't drop a + * statistics entry for another slot with the same name just created in + * another session. */ if (SlotIsLogical(slot)) - pgstat_drop_replslot(NameStr(slot->data.name)); + pgstat_drop_replslot(slot); /* * We release this at the very end, so that nobody starts trying to create diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index d73a40c1bc6..f80f90ac3cc 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -2151,7 +2151,7 @@ BufferSync(int flags) if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN) { TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id); - PendingCheckpointerStats.m_buf_written_checkpoints++; + PendingCheckpointerStats.buf_written_checkpoints++; num_written++; } } @@ -2261,7 +2261,7 @@ BgBufferSync(WritebackContext *wb_context) strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc); /* Report buffer alloc counts to pgstat */ - PendingBgWriterStats.m_buf_alloc += recent_alloc; + PendingBgWriterStats.buf_alloc += recent_alloc; /* * If we're not running the LRU scan, just stop after doing the stats @@ -2451,7 +2451,7 @@ BgBufferSync(WritebackContext *wb_context) reusable_buffers++; if (++num_written >= bgwriter_lru_maxpages) { - PendingBgWriterStats.m_maxwritten_clean++; + PendingBgWriterStats.maxwritten_clean++; break; } } @@ -2459,7 +2459,7 @@ BgBufferSync(WritebackContext *wb_context) reusable_buffers++; } - PendingBgWriterStats.m_buf_written_clean += num_written; + PendingBgWriterStats.buf_written_clean += num_written; #ifdef BGW_DEBUG elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d", diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index cd4ebe2fc5e..88ff59c568f 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -145,6 +145,7 @@ CalculateShmemSize(int *num_semaphores) size = add_size(size, BTreeShmemSize()); size = add_size(size, SyncScanShmemSize()); size = add_size(size, AsyncShmemSize()); + size = add_size(size, StatsShmemSize()); #ifdef EXEC_BACKEND size = add_size(size, ShmemBackendArraySize()); #endif @@ -296,6 +297,7 @@ CreateSharedMemoryAndSemaphores(void) BTreeShmemInit(); SyncScanShmemInit(); AsyncShmemInit(); + StatsShmemInit(); #ifdef EXEC_BACKEND diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c index 8f7f1b2f7c3..c24779d0bbd 100644 --- a/src/backend/storage/lmgr/lwlock.c +++ b/src/backend/storage/lmgr/lwlock.c @@ -176,7 +176,13 @@ static const char *const BuiltinTrancheNames[] = { /* LWTRANCHE_PARALLEL_APPEND: */ "ParallelAppend", /* LWTRANCHE_PER_XACT_PREDICATE_LIST: */ - "PerXactPredicateList" + "PerXactPredicateList", + /* LWTRANCHE_PGSTATS_DSA: */ + "PgStatsDSA", + /* LWTRANCHE_PGSTATS_HASH: */ + "PgStatsHash", + /* LWTRANCHE_PGSTATS_DATA: */ + "PgStatsData", }; StaticAssertDecl(lengthof(BuiltinTrancheNames) == diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 260b650f15f..2689d0962cf 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3380,6 +3380,12 @@ ProcessInterrupts(void) if (LogMemoryContextPending) ProcessLogMemoryContextInterrupt(); + + if (IdleStatsUpdateTimeoutPending) + { + IdleStatsUpdateTimeoutPending = false; + pgstat_report_stat(true); + } } @@ -4044,6 +4050,7 @@ PostgresMain(const char *dbname, const char *username) volatile bool send_ready_for_query = true; bool idle_in_transaction_timeout_enabled = false; bool idle_session_timeout_enabled = false; + bool idle_stats_update_timeout_enabled = false; AssertArg(dbname != NULL); AssertArg(username != NULL); @@ -4407,6 +4414,8 @@ PostgresMain(const char *dbname, const char *username) } else { + long stats_timeout; + /* * Process incoming notifies (including self-notifies), if * any, and send relevant messages to the client. Doing it @@ -4417,7 +4426,14 @@ PostgresMain(const char *dbname, const char *username) if (notifyInterruptPending) ProcessNotifyInterrupt(false); - pgstat_report_stat(false); + /* Start the idle-stats-update timer */ + stats_timeout = pgstat_report_stat(false); + if (stats_timeout > 0) + { + idle_stats_update_timeout_enabled = true; + enable_timeout_after(IDLE_STATS_UPDATE_TIMEOUT, + stats_timeout); + } set_ps_display("idle"); pgstat_report_activity(STATE_IDLE, NULL); @@ -4452,9 +4468,9 @@ PostgresMain(const char *dbname, const char *username) firstchar = ReadCommand(&input_message); /* - * (4) turn off the idle-in-transaction and idle-session timeouts, if - * active. We do this before step (5) so that any last-moment timeout - * is certain to be detected in step (5). + * (4) turn off the idle-in-transaction, idle-session and + * idle-state-update timeouts if active. We do this before step (5) so + * that any last-moment timeout is certain to be detected in step (5). * * At most one of these timeouts will be active, so there's no need to * worry about combining the timeout.c calls into one. @@ -4469,6 +4485,11 @@ PostgresMain(const char *dbname, const char *username) disable_timeout(IDLE_SESSION_TIMEOUT, false); idle_session_timeout_enabled = false; } + if (idle_stats_update_timeout_enabled) + { + disable_timeout(IDLE_STATS_UPDATE_TIMEOUT, false); + idle_stats_update_timeout_enabled = false; + } /* * (5) disable async signal conditions again. diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile index 791ba68e7e3..690312308f5 100644 --- a/src/backend/utils/activity/Makefile +++ b/src/backend/utils/activity/Makefile @@ -23,6 +23,7 @@ OBJS = \ pgstat_function.o \ pgstat_relation.o \ pgstat_replslot.o \ + pgstat_shmem.o \ pgstat_slru.o \ pgstat_subscription.o \ pgstat_wal.o \ diff --git a/src/backend/utils/activity/pgstat_archiver.c b/src/backend/utils/activity/pgstat_archiver.c index 09bc12070da..65f66dd9b31 100644 --- a/src/backend/utils/activity/pgstat_archiver.c +++ b/src/backend/utils/activity/pgstat_archiver.c @@ -27,14 +27,85 @@ void pgstat_report_archiver(const char *xlog, bool failed) { - PgStat_MsgArchiver msg; + PgStatShared_Archiver *stats_shmem = &pgStatLocal.shmem->archiver; + TimestampTz now = GetCurrentTimestamp(); - /* - * Prepare and send the message - */ - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_ARCHIVER); - msg.m_failed = failed; - strlcpy(msg.m_xlog, xlog, sizeof(msg.m_xlog)); - msg.m_timestamp = GetCurrentTimestamp(); - pgstat_send(&msg, sizeof(msg)); + changecount_before_write(&stats_shmem->changecount); + + if (failed) + { + ++stats_shmem->stats.failed_count; + memcpy(&stats_shmem->stats.last_failed_wal, xlog, + sizeof(stats_shmem->stats.last_failed_wal)); + stats_shmem->stats.last_failed_timestamp = now; + } + else + { + ++stats_shmem->stats.archived_count; + memcpy(&stats_shmem->stats.last_archived_wal, xlog, + sizeof(stats_shmem->stats.last_archived_wal)); + stats_shmem->stats.last_archived_timestamp = now; + } + + changecount_after_write(&stats_shmem->changecount); +} + +/* + * Support function for the SQL-callable pgstat* functions. Returns + * a pointer to the archiver statistics struct. + */ +PgStat_ArchiverStats * +pgstat_fetch_stat_archiver(void) +{ + pgstat_snapshot_global(PGSTAT_KIND_ARCHIVER); + + return &pgStatLocal.snapshot.archiver; +} + +void +pgstat_archiver_reset_all_cb(TimestampTz now) +{ + PgStatShared_Archiver *stats_shmem = &pgStatLocal.shmem->archiver; + + /* see explanation above PgStatShared_Archiver for the reset protocol */ + LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE); + pgstat_copy_changecounted_stats(&stats_shmem->reset_offset, + &stats_shmem->stats, + sizeof(stats_shmem->stats), + &stats_shmem->changecount); + stats_shmem->stats.stat_reset_timestamp = now; + LWLockRelease(&stats_shmem->lock); +} + +void +pgstat_archiver_snapshot_cb(void) +{ + PgStatShared_Archiver *stats_shmem = &pgStatLocal.shmem->archiver; + PgStat_ArchiverStats *stat_snap = &pgStatLocal.snapshot.archiver; + PgStat_ArchiverStats *reset_offset = &stats_shmem->reset_offset; + PgStat_ArchiverStats reset; + + pgstat_copy_changecounted_stats(stat_snap, + &stats_shmem->stats, + sizeof(stats_shmem->stats), + &stats_shmem->changecount); + + LWLockAcquire(&stats_shmem->lock, LW_SHARED); + memcpy(&reset, reset_offset, sizeof(stats_shmem->stats)); + LWLockRelease(&stats_shmem->lock); + + /* compensate by reset offsets */ + if (stat_snap->archived_count == reset.archived_count) + { + stat_snap->last_archived_wal[0] = 0; + stat_snap->last_archived_timestamp = 0; + } + stat_snap->archived_count -= reset.archived_count; + + if (stat_snap->failed_count == reset.failed_count) + { + stat_snap->last_failed_wal[0] = 0; + stat_snap->last_failed_timestamp = 0; + } + stat_snap->failed_count -= reset.failed_count; } diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index dfea88eca10..614acda1a05 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -20,12 +20,7 @@ #include "utils/pgstat_internal.h" -/* - * BgWriter global statistics counters. Stored directly in a stats - * message structure so they can be sent without needing to copy things - * around. We assume this init to zeroes. - */ -PgStat_MsgBgWriter PendingBgWriterStats; +PgStat_BgWriterStats PendingBgWriterStats = {0}; /* @@ -34,27 +29,81 @@ PgStat_MsgBgWriter PendingBgWriterStats; void pgstat_report_bgwriter(void) { - /* We assume this initializes to zeroes */ - static const PgStat_MsgBgWriter all_zeroes; + static const PgStat_BgWriterStats all_zeroes; + Assert(!pgStatLocal.shmem->is_shutdown); pgstat_assert_is_up(); /* * This function can be called even if nothing at all has happened. In - * this case, avoid sending a completely empty message to the stats - * collector. + * this case, avoid unnecessarily modifying the stats entry. */ - if (memcmp(&PendingBgWriterStats, &all_zeroes, sizeof(PgStat_MsgBgWriter)) == 0) + if (memcmp(&PendingBgWriterStats, &all_zeroes, sizeof(all_zeroes)) == 0) return; - /* - * Prepare and send the message - */ - pgstat_setheader(&PendingBgWriterStats.m_hdr, PGSTAT_MTYPE_BGWRITER); - pgstat_send(&PendingBgWriterStats, sizeof(PendingBgWriterStats)); + changecount_before_write(&pgStatLocal.shmem->bgwriter.changecount); + +#define BGWRITER_ACC(fld) pgStatLocal.shmem->bgwriter.stats.fld += PendingBgWriterStats.fld + BGWRITER_ACC(buf_written_clean); + BGWRITER_ACC(maxwritten_clean); + BGWRITER_ACC(buf_alloc); +#undef BGWRITER_ACC + + changecount_after_write(&pgStatLocal.shmem->bgwriter.changecount); /* * Clear out the statistics buffer, so it can be re-used. */ MemSet(&PendingBgWriterStats, 0, sizeof(PendingBgWriterStats)); } + +/* + * Support function for the SQL-callable pgstat* functions. Returns + * a pointer to the bgwriter statistics struct. + */ +PgStat_BgWriterStats * +pgstat_fetch_stat_bgwriter(void) +{ + pgstat_snapshot_global(PGSTAT_KIND_BGWRITER); + + return &pgStatLocal.snapshot.bgwriter; +} + +void +pgstat_bgwriter_reset_all_cb(TimestampTz now) +{ + PgStatShared_BgWriter *stats_shmem = &pgStatLocal.shmem->bgwriter; + + /* see explanation above PgStatShared_BgWriter for the reset protocol */ + LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE); + pgstat_copy_changecounted_stats(&stats_shmem->reset_offset, + &stats_shmem->stats, + sizeof(stats_shmem->stats), + &stats_shmem->changecount); + stats_shmem->stats.stat_reset_timestamp = now; + LWLockRelease(&stats_shmem->lock); +} + +void +pgstat_bgwriter_snapshot_cb(void) +{ + PgStatShared_BgWriter *stats_shmem = &pgStatLocal.shmem->bgwriter; + PgStat_BgWriterStats *reset_offset = &stats_shmem->reset_offset; + PgStat_BgWriterStats reset; + + pgstat_copy_changecounted_stats(&pgStatLocal.snapshot.bgwriter, + &stats_shmem->stats, + sizeof(stats_shmem->stats), + &stats_shmem->changecount); + + LWLockAcquire(&stats_shmem->lock, LW_SHARED); + memcpy(&reset, reset_offset, sizeof(stats_shmem->stats)); + LWLockRelease(&stats_shmem->lock); + + /* compensate by reset offsets */ +#define BGWRITER_COMP(fld) pgStatLocal.snapshot.bgwriter.fld -= reset.fld; + BGWRITER_COMP(buf_written_clean); + BGWRITER_COMP(maxwritten_clean); + BGWRITER_COMP(buf_alloc); +#undef BGWRITER_COMP +} diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 3f4e2054f55..411dbf8c374 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -20,12 +20,7 @@ #include "utils/pgstat_internal.h" -/* - * Checkpointer global statistics counters. Stored directly in a stats - * message structure so they can be sent without needing to copy things - * around. We assume this init to zeroes. - */ -PgStat_MsgCheckpointer PendingCheckpointerStats; +PgStat_CheckpointerStats PendingCheckpointerStats = {0}; /* @@ -35,24 +30,92 @@ void pgstat_report_checkpointer(void) { /* We assume this initializes to zeroes */ - static const PgStat_MsgCheckpointer all_zeroes; + static const PgStat_CheckpointerStats all_zeroes; + PgStatShared_Checkpointer *stats_shmem = &pgStatLocal.shmem->checkpointer; + + Assert(!pgStatLocal.shmem->is_shutdown); + pgstat_assert_is_up(); /* * This function can be called even if nothing at all has happened. In - * this case, avoid sending a completely empty message to the stats - * collector. + * this case, avoid unnecessarily modifying the stats entry. */ - if (memcmp(&PendingCheckpointerStats, &all_zeroes, sizeof(PgStat_MsgCheckpointer)) == 0) + if (memcmp(&PendingCheckpointerStats, &all_zeroes, + sizeof(all_zeroes)) == 0) return; - /* - * Prepare and send the message - */ - pgstat_setheader(&PendingCheckpointerStats.m_hdr, PGSTAT_MTYPE_CHECKPOINTER); - pgstat_send(&PendingCheckpointerStats, sizeof(PendingCheckpointerStats)); + changecount_before_write(&stats_shmem->changecount); + +#define CHECKPOINTER_ACC(fld) stats_shmem->stats.fld += PendingCheckpointerStats.fld + CHECKPOINTER_ACC(timed_checkpoints); + CHECKPOINTER_ACC(requested_checkpoints); + CHECKPOINTER_ACC(checkpoint_write_time); + CHECKPOINTER_ACC(checkpoint_sync_time); + CHECKPOINTER_ACC(buf_written_checkpoints); + CHECKPOINTER_ACC(buf_written_backend); + CHECKPOINTER_ACC(buf_fsync_backend); +#undef CHECKPOINTER_ACC + + changecount_after_write(&stats_shmem->changecount); /* * Clear out the statistics buffer, so it can be re-used. */ MemSet(&PendingCheckpointerStats, 0, sizeof(PendingCheckpointerStats)); } + +/* + * pgstat_fetch_stat_checkpointer() - + * + * Support function for the SQL-callable pgstat* functions. Returns + * a pointer to the checkpointer statistics struct. + */ +PgStat_CheckpointerStats * +pgstat_fetch_stat_checkpointer(void) +{ + pgstat_snapshot_global(PGSTAT_KIND_CHECKPOINTER); + + return &pgStatLocal.snapshot.checkpointer; +} + +void +pgstat_checkpointer_reset_all_cb(TimestampTz now) +{ + PgStatShared_Checkpointer *stats_shmem = &pgStatLocal.shmem->checkpointer; + + /* see explanation above PgStatShared_Checkpointer for the reset protocol */ + LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE); + pgstat_copy_changecounted_stats(&stats_shmem->reset_offset, + &stats_shmem->stats, + sizeof(stats_shmem->stats), + &stats_shmem->changecount); + LWLockRelease(&stats_shmem->lock); +} + +void +pgstat_checkpointer_snapshot_cb(void) +{ + PgStatShared_Checkpointer *stats_shmem = &pgStatLocal.shmem->checkpointer; + PgStat_CheckpointerStats *reset_offset = &stats_shmem->reset_offset; + PgStat_CheckpointerStats reset; + + pgstat_copy_changecounted_stats(&pgStatLocal.snapshot.checkpointer, + &stats_shmem->stats, + sizeof(stats_shmem->stats), + &stats_shmem->changecount); + + LWLockAcquire(&stats_shmem->lock, LW_SHARED); + memcpy(&reset, reset_offset, sizeof(stats_shmem->stats)); + LWLockRelease(&stats_shmem->lock); + + /* compensate by reset offsets */ +#define CHECKPOINTER_COMP(fld) pgStatLocal.snapshot.checkpointer.fld -= reset.fld; + CHECKPOINTER_COMP(timed_checkpoints); + CHECKPOINTER_COMP(requested_checkpoints); + CHECKPOINTER_COMP(checkpoint_write_time); + CHECKPOINTER_COMP(checkpoint_sync_time); + CHECKPOINTER_COMP(buf_written_checkpoints); + CHECKPOINTER_COMP(buf_written_backend); + CHECKPOINTER_COMP(buf_fsync_backend); +#undef CHECKPOINTER_COMP +} diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 0f2dd956d36..2d1b4b401e1 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -19,39 +19,30 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "storage/procsignal.h" static bool pgstat_should_report_connstat(void); -int pgStatXactCommit = 0; -int pgStatXactRollback = 0; PgStat_Counter pgStatBlockReadTime = 0; PgStat_Counter pgStatBlockWriteTime = 0; PgStat_Counter pgStatActiveTime = 0; PgStat_Counter pgStatTransactionIdleTime = 0; SessionEndType pgStatSessionEndCause = DISCONNECT_NORMAL; - +static int pgStatXactCommit = 0; +static int pgStatXactRollback = 0; static PgStat_Counter pgLastSessionReportTime = 0; /* - * Tell the collector that we just dropped a database. - * (If the message gets lost, we will still clean the dead DB eventually - * via future invocations of pgstat_vacuum_stat().) + * Remove entry for the database being dropped. */ void pgstat_drop_database(Oid databaseid) { - PgStat_MsgDropdb msg; - - if (pgStatSock == PGINVALID_SOCKET) - return; - - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_DROPDB); - msg.m_databaseid = databaseid; - pgstat_send(&msg, sizeof(msg)); + pgstat_drop_transactional(PGSTAT_KIND_DATABASE, databaseid, InvalidOid); } /* @@ -60,15 +51,39 @@ pgstat_drop_database(Oid databaseid) void pgstat_report_recovery_conflict(int reason) { - PgStat_MsgRecoveryConflict msg; + PgStat_StatDBEntry *dbentry; - if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts) + Assert(IsUnderPostmaster); + if (!pgstat_track_counts) return; - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RECOVERYCONFLICT); - msg.m_databaseid = MyDatabaseId; - msg.m_reason = reason; - pgstat_send(&msg, sizeof(msg)); + dbentry = pgstat_prep_database_pending(MyDatabaseId); + + switch (reason) + { + case PROCSIG_RECOVERY_CONFLICT_DATABASE: + + /* + * Since we drop the information about the database as soon as it + * replicates, there is no point in counting these conflicts. + */ + break; + case PROCSIG_RECOVERY_CONFLICT_TABLESPACE: + dbentry->n_conflict_tablespace++; + break; + case PROCSIG_RECOVERY_CONFLICT_LOCK: + dbentry->n_conflict_lock++; + break; + case PROCSIG_RECOVERY_CONFLICT_SNAPSHOT: + dbentry->n_conflict_snapshot++; + break; + case PROCSIG_RECOVERY_CONFLICT_BUFFERPIN: + dbentry->n_conflict_bufferpin++; + break; + case PROCSIG_RECOVERY_CONFLICT_STARTUP_DEADLOCK: + dbentry->n_conflict_startup_deadlock++; + break; + } } /* @@ -77,14 +92,13 @@ pgstat_report_recovery_conflict(int reason) void pgstat_report_deadlock(void) { - PgStat_MsgDeadlock msg; + PgStat_StatDBEntry *dbent; - if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts) + if (!pgstat_track_counts) return; - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_DEADLOCK); - msg.m_databaseid = MyDatabaseId; - pgstat_send(&msg, sizeof(msg)); + dbent = pgstat_prep_database_pending(MyDatabaseId); + dbent->n_deadlocks++; } /* @@ -93,17 +107,24 @@ pgstat_report_deadlock(void) void pgstat_report_checksum_failures_in_db(Oid dboid, int failurecount) { - PgStat_MsgChecksumFailure msg; + PgStat_EntryRef *entry_ref; + PgStatShared_Database *sharedent; - if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts) + if (!pgstat_track_counts) return; - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_CHECKSUMFAILURE); - msg.m_databaseid = dboid; - msg.m_failurecount = failurecount; - msg.m_failure_time = GetCurrentTimestamp(); + /* + * Update the shared stats directly - checksum failures should never be + * common enough for that to be a problem. + */ + entry_ref = + pgstat_get_entry_ref_locked(PGSTAT_KIND_DATABASE, dboid, InvalidOid, false); - pgstat_send(&msg, sizeof(msg)); + sharedent = (PgStatShared_Database *) entry_ref->shared_stats; + sharedent->stats.n_checksum_failures += failurecount; + sharedent->stats.last_checksum_failure = GetCurrentTimestamp(); + + pgstat_unlock_entry(entry_ref); } /* @@ -121,15 +142,17 @@ pgstat_report_checksum_failure(void) void pgstat_report_tempfile(size_t filesize) { - PgStat_MsgTempFile msg; + PgStat_StatDBEntry *dbent; - if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts) + if (!pgstat_track_counts) return; - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_TEMPFILE); - msg.m_databaseid = MyDatabaseId; - msg.m_filesize = filesize; - pgstat_send(&msg, sizeof(msg)); + if (filesize == 0) /* Is there a case where filesize is really 0? */ + return; + + dbent = pgstat_prep_database_pending(MyDatabaseId); + dbent->n_temp_bytes += filesize; /* needs check overflow */ + dbent->n_temp_files++; } /* @@ -138,16 +161,15 @@ pgstat_report_tempfile(size_t filesize) void pgstat_report_connect(Oid dboid) { - PgStat_MsgConnect msg; + PgStat_StatDBEntry *dbentry; if (!pgstat_should_report_connstat()) return; pgLastSessionReportTime = MyStartTimestamp; - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_CONNECT); - msg.m_databaseid = MyDatabaseId; - pgstat_send(&msg, sizeof(PgStat_MsgConnect)); + dbentry = pgstat_prep_database_pending(MyDatabaseId); + dbentry->n_sessions++; } /* @@ -156,15 +178,42 @@ pgstat_report_connect(Oid dboid) void pgstat_report_disconnect(Oid dboid) { - PgStat_MsgDisconnect msg; + PgStat_StatDBEntry *dbentry; if (!pgstat_should_report_connstat()) return; - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_DISCONNECT); - msg.m_databaseid = MyDatabaseId; - msg.m_cause = pgStatSessionEndCause; - pgstat_send(&msg, sizeof(PgStat_MsgDisconnect)); + dbentry = pgstat_prep_database_pending(MyDatabaseId); + + switch (pgStatSessionEndCause) + { + case DISCONNECT_NOT_YET: + case DISCONNECT_NORMAL: + /* we don't collect these */ + break; + case DISCONNECT_CLIENT_EOF: + dbentry->n_sessions_abandoned++; + break; + case DISCONNECT_FATAL: + dbentry->n_sessions_fatal++; + break; + case DISCONNECT_KILLED: + dbentry->n_sessions_killed++; + break; + } +} + +/* + * Support function for the SQL-callable pgstat* functions. Returns + * the collected statistics for one database or NULL. NULL doesn't mean + * that the database doesn't exist, just that there are no statistics, so the + * caller is better off to report ZERO instead. + */ +PgStat_StatDBEntry * +pgstat_fetch_stat_dbentry(Oid dboid) +{ + return (PgStat_StatDBEntry *) + pgstat_fetch_entry(PGSTAT_KIND_DATABASE, dboid, InvalidOid); } void @@ -185,57 +234,47 @@ AtEOXact_PgStat_Database(bool isCommit, bool parallel) } /* - * Subroutine for pgstat_send_tabstat: Handle xact commit/rollback and I/O + * Subroutine for pgstat_report_stat(): Handle xact commit/rollback and I/O * timings. */ void -pgstat_update_dbstats(PgStat_MsgTabstat *tsmsg, TimestampTz now) +pgstat_update_dbstats(TimestampTz now) { - if (OidIsValid(tsmsg->m_databaseid)) - { - tsmsg->m_xact_commit = pgStatXactCommit; - tsmsg->m_xact_rollback = pgStatXactRollback; - tsmsg->m_block_read_time = pgStatBlockReadTime; - tsmsg->m_block_write_time = pgStatBlockWriteTime; + PgStat_StatDBEntry *dbentry; - if (pgstat_should_report_connstat()) - { - long secs; - int usecs; + dbentry = pgstat_prep_database_pending(MyDatabaseId); - /* - * pgLastSessionReportTime is initialized to MyStartTimestamp by - * pgstat_report_connect(). - */ - TimestampDifference(pgLastSessionReportTime, now, &secs, &usecs); - pgLastSessionReportTime = now; - tsmsg->m_session_time = (PgStat_Counter) secs * 1000000 + usecs; - tsmsg->m_active_time = pgStatActiveTime; - tsmsg->m_idle_in_xact_time = pgStatTransactionIdleTime; - } - else - { - tsmsg->m_session_time = 0; - tsmsg->m_active_time = 0; - tsmsg->m_idle_in_xact_time = 0; - } - pgStatXactCommit = 0; - pgStatXactRollback = 0; - pgStatBlockReadTime = 0; - pgStatBlockWriteTime = 0; - pgStatActiveTime = 0; - pgStatTransactionIdleTime = 0; - } - else + /* + * Accumulate xact commit/rollback and I/O timings to stats entry of the + * current database. + */ + dbentry->n_xact_commit += pgStatXactCommit; + dbentry->n_xact_rollback += pgStatXactRollback; + dbentry->n_block_read_time += pgStatBlockReadTime; + dbentry->n_block_write_time += pgStatBlockWriteTime; + + if (pgstat_should_report_connstat()) { - tsmsg->m_xact_commit = 0; - tsmsg->m_xact_rollback = 0; - tsmsg->m_block_read_time = 0; - tsmsg->m_block_write_time = 0; - tsmsg->m_session_time = 0; - tsmsg->m_active_time = 0; - tsmsg->m_idle_in_xact_time = 0; + long secs; + int usecs; + + /* + * pgLastSessionReportTime is initialized to MyStartTimestamp by + * pgstat_report_connect(). + */ + TimestampDifference(pgLastSessionReportTime, now, &secs, &usecs); + pgLastSessionReportTime = now; + dbentry->total_session_time += (PgStat_Counter) secs * 1000000 + usecs; + dbentry->total_active_time += pgStatActiveTime; + dbentry->total_idle_in_xact_time += pgStatTransactionIdleTime; } + + pgStatXactCommit = 0; + pgStatXactRollback = 0; + pgStatBlockReadTime = 0; + pgStatBlockWriteTime = 0; + pgStatActiveTime = 0; + pgStatTransactionIdleTime = 0; } /* @@ -250,3 +289,111 @@ pgstat_should_report_connstat(void) { return MyBackendType == B_BACKEND; } + +/* + * Find or create a local PgStat_StatDBEntry entry for dboid. + */ +PgStat_StatDBEntry * +pgstat_prep_database_pending(Oid dboid) +{ + PgStat_EntryRef *entry_ref; + + entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_DATABASE, dboid, InvalidOid, + NULL); + + return entry_ref->pending; + +} + +/* + * Reset the database's reset timestamp, without resetting the contents of the + * database stats. + */ +void +pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) +{ + PgStat_EntryRef *dbref; + PgStatShared_Database *dbentry; + + dbref = pgstat_get_entry_ref_locked(PGSTAT_KIND_DATABASE, MyDatabaseId, InvalidOid, + false); + + dbentry = (PgStatShared_Database *) dbref->shared_stats; + dbentry->stats.stat_reset_timestamp = ts; + + pgstat_unlock_entry(dbref); +} + +/* + * Flush out pending stats for the entry + * + * If nowait is true, this function returns false if lock could not + * immediately acquired, otherwise true is returned. + */ +bool +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +{ + PgStatShared_Database *sharedent; + PgStat_StatDBEntry *pendingent; + + pendingent = (PgStat_StatDBEntry *) entry_ref->pending; + sharedent = (PgStatShared_Database *) entry_ref->shared_stats; + + if (!pgstat_lock_entry(entry_ref, nowait)) + return false; + +#define PGSTAT_ACCUM_DBCOUNT(item) \ + (sharedent)->stats.item += (pendingent)->item + + PGSTAT_ACCUM_DBCOUNT(n_xact_commit); + PGSTAT_ACCUM_DBCOUNT(n_xact_rollback); + PGSTAT_ACCUM_DBCOUNT(n_blocks_fetched); + PGSTAT_ACCUM_DBCOUNT(n_blocks_hit); + + PGSTAT_ACCUM_DBCOUNT(n_tuples_returned); + PGSTAT_ACCUM_DBCOUNT(n_tuples_fetched); + PGSTAT_ACCUM_DBCOUNT(n_tuples_inserted); + PGSTAT_ACCUM_DBCOUNT(n_tuples_updated); + PGSTAT_ACCUM_DBCOUNT(n_tuples_deleted); + + /* last_autovac_time is reported immediately */ + Assert(pendingent->last_autovac_time == 0); + + PGSTAT_ACCUM_DBCOUNT(n_conflict_tablespace); + PGSTAT_ACCUM_DBCOUNT(n_conflict_lock); + PGSTAT_ACCUM_DBCOUNT(n_conflict_snapshot); + PGSTAT_ACCUM_DBCOUNT(n_conflict_bufferpin); + PGSTAT_ACCUM_DBCOUNT(n_conflict_startup_deadlock); + + PGSTAT_ACCUM_DBCOUNT(n_temp_bytes); + PGSTAT_ACCUM_DBCOUNT(n_temp_files); + PGSTAT_ACCUM_DBCOUNT(n_deadlocks); + + /* checksum failures are reported immediately */ + Assert(pendingent->n_checksum_failures == 0); + Assert(pendingent->last_checksum_failure == 0); + + PGSTAT_ACCUM_DBCOUNT(n_block_read_time); + PGSTAT_ACCUM_DBCOUNT(n_block_write_time); + + PGSTAT_ACCUM_DBCOUNT(n_sessions); + PGSTAT_ACCUM_DBCOUNT(total_session_time); + PGSTAT_ACCUM_DBCOUNT(total_active_time); + PGSTAT_ACCUM_DBCOUNT(total_idle_in_xact_time); + PGSTAT_ACCUM_DBCOUNT(n_sessions_abandoned); + PGSTAT_ACCUM_DBCOUNT(n_sessions_fatal); + PGSTAT_ACCUM_DBCOUNT(n_sessions_killed); +#undef PGSTAT_ACCUM_DBCOUNT + + pgstat_unlock_entry(entry_ref); + + memset(pendingent, 0, sizeof(*pendingent)); + + return true; +} + +void +pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts) +{ + ((PgStatShared_Database *) header)->stats.stat_reset_timestamp = ts; +} diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index ad9879afb2a..81d2d18b1f6 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -17,7 +17,9 @@ #include "postgres.h" +#include "utils/inval.h" #include "utils/pgstat_internal.h" +#include "utils/syscache.h" #include "utils/timestamp.h" @@ -28,18 +30,6 @@ int pgstat_track_functions = TRACK_FUNC_OFF; -/* - * Indicates if backend has some function stats that it hasn't yet - * sent to the collector. - */ -bool have_function_stats = false; - -/* - * Backends store per-function info that's waiting to be sent to the collector - * in this hash table (indexed by function OID). - */ -static HTAB *pgStatFunctions = NULL; - /* * Total time charged to functions so far in the current backend. * We use this to help separate "self" and "other" time charges. @@ -61,6 +51,10 @@ pgstat_create_function(Oid proid) /* * Ensure that stats are dropped if transaction commits. + * + * NB: This is only reliable because pgstat_init_function_usage() does some + * extra work. If other places start emitting function stats they likely need + * similar logic. */ void pgstat_drop_function(Oid proid) @@ -78,8 +72,9 @@ void pgstat_init_function_usage(FunctionCallInfo fcinfo, PgStat_FunctionCallUsage *fcu) { - PgStat_BackendFunctionEntry *htabent; - bool found; + PgStat_EntryRef *entry_ref; + PgStat_BackendFunctionEntry *pending; + bool created_entry; if (pgstat_track_functions <= fcinfo->flinfo->fn_stats) { @@ -88,29 +83,48 @@ pgstat_init_function_usage(FunctionCallInfo fcinfo, return; } - if (!pgStatFunctions) - { - /* First time through - initialize function stat table */ - HASHCTL hash_ctl; + entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_FUNCTION, + MyDatabaseId, + fcinfo->flinfo->fn_oid, + &created_entry); - hash_ctl.keysize = sizeof(Oid); - hash_ctl.entrysize = sizeof(PgStat_BackendFunctionEntry); - pgStatFunctions = hash_create("Function stat entries", - PGSTAT_FUNCTION_HASH_SIZE, - &hash_ctl, - HASH_ELEM | HASH_BLOBS); + /* + * If no shared entry already exists, check if the function has been + * deleted concurrently. This can go unnoticed until here because + * executing a statement that just calls a function, does not trigger + * cache invalidation processing. The reason we care about this case is + * that otherwise we could create a new stats entry for an already dropped + * function (for relations etc this is not possible because emitting stats + * requires a lock for the relation to already have been acquired). + * + * It's somewhat ugly to have a behavioural difference based on + * track_functions being enabled/disabled. But it seems acceptable, given + * that there's already behavioural differences depending on whether the + * function is the caches etc. + * + * For correctness it'd be sufficient to set ->dropped to true. However, + * the accepted invalidation will commonly cause "low level" failures in + * PL code, with an OID in the error message. Making this harder to + * test... + */ + if (created_entry) + { + AcceptInvalidationMessages(); + if (!SearchSysCacheExists1(PROCOID, ObjectIdGetDatum(fcinfo->flinfo->fn_oid))) + { + pgstat_drop_entry(PGSTAT_KIND_FUNCTION, MyDatabaseId, + fcinfo->flinfo->fn_oid); + ereport(ERROR, errcode(ERRCODE_UNDEFINED_FUNCTION), + errmsg("function call to dropped function")); + } } - /* Get the stats entry for this function, create if necessary */ - htabent = hash_search(pgStatFunctions, &fcinfo->flinfo->fn_oid, - HASH_ENTER, &found); - if (!found) - MemSet(&htabent->f_counts, 0, sizeof(PgStat_FunctionCounts)); + pending = entry_ref->pending; - fcu->fs = &htabent->f_counts; + fcu->fs = &pending->f_counts; /* save stats for this function, later used to compensate for recursion */ - fcu->save_f_total_time = htabent->f_counts.f_total_time; + fcu->save_f_total_time = pending->f_counts.f_total_time; /* save current backend-wide total time */ fcu->save_total = total_func_time; @@ -167,64 +181,37 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) fs->f_numcalls++; fs->f_total_time = f_total; INSTR_TIME_ADD(fs->f_self_time, f_self); - - /* indicate that we have something to send */ - have_function_stats = true; } /* - * Subroutine for pgstat_report_stat: populate and send a function stat message + * Flush out pending stats for the entry + * + * If nowait is true, this function returns false if lock could not + * immediately acquired, otherwise true is returned. */ -void -pgstat_send_funcstats(void) +bool +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) { - /* we assume this inits to all zeroes: */ - static const PgStat_FunctionCounts all_zeroes; + PgStat_BackendFunctionEntry *localent; + PgStatShared_Function *shfuncent; - PgStat_MsgFuncstat msg; - PgStat_BackendFunctionEntry *entry; - HASH_SEQ_STATUS fstat; + localent = (PgStat_BackendFunctionEntry *) entry_ref->pending; + shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; - if (pgStatFunctions == NULL) - return; + /* localent always has non-zero content */ - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_FUNCSTAT); - msg.m_databaseid = MyDatabaseId; - msg.m_nentries = 0; + if (!pgstat_lock_entry(entry_ref, nowait)) + return false; - hash_seq_init(&fstat, pgStatFunctions); - while ((entry = (PgStat_BackendFunctionEntry *) hash_seq_search(&fstat)) != NULL) - { - PgStat_FunctionEntry *m_ent; + shfuncent->stats.f_numcalls += localent->f_counts.f_numcalls; + shfuncent->stats.f_total_time += + INSTR_TIME_GET_MICROSEC(localent->f_counts.f_total_time); + shfuncent->stats.f_self_time += + INSTR_TIME_GET_MICROSEC(localent->f_counts.f_self_time); - /* Skip it if no counts accumulated since last time */ - if (memcmp(&entry->f_counts, &all_zeroes, - sizeof(PgStat_FunctionCounts)) == 0) - continue; + pgstat_unlock_entry(entry_ref); - /* need to convert format of time accumulators */ - m_ent = &msg.m_entry[msg.m_nentries]; - m_ent->f_id = entry->f_id; - m_ent->f_numcalls = entry->f_counts.f_numcalls; - m_ent->f_total_time = INSTR_TIME_GET_MICROSEC(entry->f_counts.f_total_time); - m_ent->f_self_time = INSTR_TIME_GET_MICROSEC(entry->f_counts.f_self_time); - - if (++msg.m_nentries >= PGSTAT_NUM_FUNCENTRIES) - { - pgstat_send(&msg, offsetof(PgStat_MsgFuncstat, m_entry[0]) + - msg.m_nentries * sizeof(PgStat_FunctionEntry)); - msg.m_nentries = 0; - } - - /* reset the entry's counts */ - MemSet(&entry->f_counts, 0, sizeof(PgStat_FunctionCounts)); - } - - if (msg.m_nentries > 0) - pgstat_send(&msg, offsetof(PgStat_MsgFuncstat, m_entry[0]) + - msg.m_nentries * sizeof(PgStat_FunctionEntry)); - - have_function_stats = false; + return true; } /* @@ -235,12 +222,22 @@ pgstat_send_funcstats(void) PgStat_BackendFunctionEntry * find_funcstat_entry(Oid func_id) { - pgstat_assert_is_up(); + PgStat_EntryRef *entry_ref; - if (pgStatFunctions == NULL) - return NULL; + entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_FUNCTION, MyDatabaseId, func_id); - return (PgStat_BackendFunctionEntry *) hash_search(pgStatFunctions, - (void *) &func_id, - HASH_FIND, NULL); + if (entry_ref) + return entry_ref->pending; + return NULL; +} + +/* + * Support function for the SQL-callable pgstat* functions. Returns + * the collected statistics for one function or NULL. + */ +PgStat_StatFuncEntry * +pgstat_fetch_stat_funcentry(Oid func_id) +{ + return (PgStat_StatFuncEntry *) + pgstat_fetch_entry(PGSTAT_KIND_FUNCTION, MyDatabaseId, func_id); } diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index 53080eadb89..039206e303b 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -19,6 +19,7 @@ #include "access/twophase_rmgr.h" #include "access/xact.h" +#include "catalog/partition.h" #include "postmaster/autovacuum.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" @@ -26,38 +27,6 @@ #include "utils/timestamp.h" -/* - * Structures in which backends store per-table info that's waiting to be - * sent to the collector. - * - * NOTE: once allocated, TabStatusArray structures are never moved or deleted - * for the life of the backend. Also, we zero out the t_id fields of the - * contained PgStat_TableStatus structs whenever they are not actively in use. - * This allows relcache pgstat_info pointers to be treated as long-lived data, - * avoiding repeated searches in pgstat_relation_init() when a relation is - * repeatedly opened during a transaction. - */ -#define TABSTAT_QUANTUM 100 /* we alloc this many at a time */ - - -typedef struct TabStatusArray -{ - struct TabStatusArray *tsa_next; /* link to next array, if any */ - int tsa_used; /* # entries currently used */ - PgStat_TableStatus tsa_entries[TABSTAT_QUANTUM]; /* per-table data */ -} TabStatusArray; - -static TabStatusArray *pgStatTabList = NULL; - -/* - * pgStatTabHash entry: map from relation OID to PgStat_TableStatus pointer - */ -typedef struct TabStatHashEntry -{ - Oid t_id; - PgStat_TableStatus *tsa_entry; -} TabStatHashEntry; - /* Record that's written to 2PC state file when pgstat state is persisted */ typedef struct TwoPhasePgStatRecord { @@ -74,27 +43,13 @@ typedef struct TwoPhasePgStatRecord } TwoPhasePgStatRecord; -static PgStat_TableStatus *get_tabstat_entry(Oid rel_id, bool isshared); -static void pgstat_send_tabstat(PgStat_MsgTabstat *tsmsg, TimestampTz now); +static PgStat_TableStatus *pgstat_prep_relation_pending(Oid rel_id, bool isshared); static void add_tabstat_xact_level(PgStat_TableStatus *pgstat_info, int nest_level); static void ensure_tabstat_xact_level(PgStat_TableStatus *pgstat_info); static void pgstat_truncdrop_save_counters(PgStat_TableXactStatus *trans, bool is_drop); static void pgstat_truncdrop_restore_counters(PgStat_TableXactStatus *trans); -/* - * Indicates if backend has some relation stats that it hasn't yet - * sent to the collector. - */ -bool have_relation_stats; - - -/* - * Hash table for O(1) t_id -> tsa_entry lookup - */ -static HTAB *pgStatTabHash = NULL; - - /* * Copy stats between relations. This is used for things like REINDEX * CONCURRENTLY. @@ -103,43 +58,44 @@ void pgstat_copy_relation_stats(Relation dst, Relation src) { PgStat_StatTabEntry *srcstats; + PgStatShared_Relation *dstshstats; + PgStat_EntryRef *dst_ref; - srcstats = pgstat_fetch_stat_tabentry(RelationGetRelid(src)); - + srcstats = pgstat_fetch_stat_tabentry_ext(src->rd_rel->relisshared, + RelationGetRelid(src)); if (!srcstats) return; - if (pgstat_relation_should_count(dst)) - { - /* - * XXX: temporarily this does not actually quite do what the name - * says, and just copy index related fields. A subsequent commit will - * do more. - */ + /* + * XXX: Is it actually correct to copy all stats here? Probably fine with + * current uses of stats for indexes, but what if we tracked bloat for + * indexes via stats? + */ + dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION, + dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId, + RelationGetRelid(dst), + false); - dst->pgstat_info->t_counts.t_numscans = srcstats->numscans; - dst->pgstat_info->t_counts.t_tuples_returned = srcstats->tuples_returned; - dst->pgstat_info->t_counts.t_tuples_fetched = srcstats->tuples_fetched; - dst->pgstat_info->t_counts.t_blocks_fetched = srcstats->blocks_fetched; - dst->pgstat_info->t_counts.t_blocks_hit = srcstats->blocks_hit; + dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats; + dstshstats->stats = *srcstats; - /* the data will be sent by the next pgstat_report_stat() call */ - } + pgstat_unlock_entry(dst_ref); } /* - * Initialize a relcache entry to count access statistics. - * Called whenever a relation is opened. + * Initialize a relcache entry to count access statistics. Called whenever a + * relation is opened. * - * We assume that a relcache entry's pgstat_info field is zeroed by - * relcache.c when the relcache entry is made; thereafter it is long-lived - * data. We can avoid repeated searches of the TabStatus arrays when the - * same relation is touched repeatedly within a transaction. + * We assume that a relcache entry's pgstat_info field is zeroed by relcache.c + * when the relcache entry is made; thereafter it is long-lived data. + * + * This does not create a reference to a stats entry in shared memory, nor + * allocate memory for the pending stats. That happens in + * pgstat_relation_assoc(). */ void pgstat_relation_init(Relation rel) { - Oid rel_id = rel->rd_id; char relkind = rel->rd_rel->relkind; /* @@ -147,27 +103,68 @@ pgstat_relation_init(Relation rel) */ if (!RELKIND_HAS_STORAGE(relkind) && relkind != RELKIND_PARTITIONED_TABLE) { + rel->pgstat_enabled = false; rel->pgstat_info = NULL; return; } - if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts) + if (!pgstat_track_counts) { + if (rel->pgstat_info) + pgstat_relation_unlink(rel); + /* We're not counting at all */ + rel->pgstat_enabled = false; rel->pgstat_info = NULL; return; } - /* - * If we already set up this relation in the current transaction, nothing - * to do. - */ - if (rel->pgstat_info != NULL && - rel->pgstat_info->t_id == rel_id) - return; + rel->pgstat_enabled = true; +} + +/* + * Prepare for statistics for this relation to be collected. + * + * This ensures we have a reference to the stats entry before stats can be + * generated. That is important because a relation drop in another connection + * could otherwise lead to the stats entry being dropped, which then later + * would get recreated when flushing stats. + * + * This is separate from pgstat_relation_init() as it is not uncommon for + * relcache entries to be opened without ever getting stats reported. + */ +void +pgstat_relation_assoc(Relation rel) +{ + Assert(rel->pgstat_enabled); + Assert(rel->pgstat_info == NULL); /* Else find or make the PgStat_TableStatus entry, and update link */ - rel->pgstat_info = get_tabstat_entry(rel_id, rel->rd_rel->relisshared); + rel->pgstat_info = pgstat_prep_relation_pending(RelationGetRelid(rel), + rel->rd_rel->relisshared); + + /* don't allow link a stats to multiple relcache entries */ + Assert(rel->pgstat_info->relation == NULL); + + /* mark this relation as the owner */ + rel->pgstat_info->relation = rel; +} + +/* + * Break the mutual link between a relcache entry and pending stats entry. + * This must be called whenever one end of the link is removed. + */ +void +pgstat_relation_unlink(Relation rel) +{ + /* remove the link to stats info if any */ + if (rel->pgstat_info == NULL) + return; + + /* link sanity check */ + Assert(rel->pgstat_info->relation == rel); + rel->pgstat_info->relation = NULL; + rel->pgstat_info = NULL; } /* @@ -187,9 +184,26 @@ pgstat_create_relation(Relation rel) void pgstat_drop_relation(Relation rel) { + int nest_level = GetCurrentTransactionNestLevel(); + PgStat_TableStatus *pgstat_info = rel->pgstat_info; + pgstat_drop_transactional(PGSTAT_KIND_RELATION, rel->rd_rel->relisshared ? InvalidOid : MyDatabaseId, RelationGetRelid(rel)); + + /* + * Transactionally set counters to 0. That ensures that accesses to + * pg_stat_xact_all_tables inside the transaction show 0. + */ + if (pgstat_info && + pgstat_info->trans != NULL && + pgstat_info->trans->nest_level == nest_level) + { + pgstat_truncdrop_save_counters(pgstat_info->trans, true); + pgstat_info->trans->tuples_inserted = 0; + pgstat_info->trans->tuples_updated = 0; + pgstat_info->trans->tuples_deleted = 0; + } } /* @@ -200,16 +214,24 @@ pgstat_drop_relation(Relation rel) void pgstat_report_autovac(Oid dboid) { - PgStat_MsgAutovacStart msg; + PgStat_EntryRef *entry_ref; + PgStatShared_Database *dbentry; - if (pgStatSock == PGINVALID_SOCKET) - return; + /* can't get here in single user mode */ + Assert(IsUnderPostmaster); - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_AUTOVAC_START); - msg.m_databaseid = dboid; - msg.m_start_time = GetCurrentTimestamp(); + /* + * End-of-vacuum is reported instantly. Report the start the same way for + * consistency. Vacuum doesn't run frequently and is a long-lasting + * operation so it doesn't matter if we get blocked here a little. + */ + entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_DATABASE, + dboid, InvalidOid, false); - pgstat_send(&msg, sizeof(msg)); + dbentry = (PgStatShared_Database *) entry_ref->shared_stats; + dbentry->stats.last_autovac_time = GetCurrentTimestamp(); + + pgstat_unlock_entry(entry_ref); } /* @@ -219,19 +241,52 @@ void pgstat_report_vacuum(Oid tableoid, bool shared, PgStat_Counter livetuples, PgStat_Counter deadtuples) { - PgStat_MsgVacuum msg; + PgStat_EntryRef *entry_ref; + PgStatShared_Relation *shtabentry; + PgStat_StatTabEntry *tabentry; + Oid dboid = (shared ? InvalidOid : MyDatabaseId); + TimestampTz ts; - if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts) + if (!pgstat_track_counts) return; - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_VACUUM); - msg.m_databaseid = shared ? InvalidOid : MyDatabaseId; - msg.m_tableoid = tableoid; - msg.m_autovacuum = IsAutoVacuumWorkerProcess(); - msg.m_vacuumtime = GetCurrentTimestamp(); - msg.m_live_tuples = livetuples; - msg.m_dead_tuples = deadtuples; - pgstat_send(&msg, sizeof(msg)); + /* Store the data in the table's hash table entry. */ + ts = GetCurrentTimestamp(); + + /* block acquiring lock for the same reason as pgstat_report_autovac() */ + entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION, + dboid, tableoid, false); + + shtabentry = (PgStatShared_Relation *) entry_ref->shared_stats; + tabentry = &shtabentry->stats; + + tabentry->n_live_tuples = livetuples; + tabentry->n_dead_tuples = deadtuples; + + /* + * It is quite possible that a non-aggressive VACUUM ended up skipping + * various pages, however, we'll zero the insert counter here regardless. + * It's currently used only to track when we need to perform an "insert" + * autovacuum, which are mainly intended to freeze newly inserted tuples. + * Zeroing this may just mean we'll not try to vacuum the table again + * until enough tuples have been inserted to trigger another insert + * autovacuum. An anti-wraparound autovacuum will catch any persistent + * stragglers. + */ + tabentry->inserts_since_vacuum = 0; + + if (IsAutoVacuumWorkerProcess()) + { + tabentry->autovac_vacuum_timestamp = ts; + tabentry->autovac_vacuum_count++; + } + else + { + tabentry->vacuum_timestamp = ts; + tabentry->vacuum_count++; + } + + pgstat_unlock_entry(entry_ref); } /* @@ -245,9 +300,12 @@ pgstat_report_analyze(Relation rel, PgStat_Counter livetuples, PgStat_Counter deadtuples, bool resetcounter) { - PgStat_MsgAnalyze msg; + PgStat_EntryRef *entry_ref; + PgStatShared_Relation *shtabentry; + PgStat_StatTabEntry *tabentry; + Oid dboid = (rel->rd_rel->relisshared ? InvalidOid : MyDatabaseId); - if (pgStatSock == PGINVALID_SOCKET || !pgstat_track_counts) + if (!pgstat_track_counts) return; /* @@ -279,15 +337,39 @@ pgstat_report_analyze(Relation rel, deadtuples = Max(deadtuples, 0); } - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_ANALYZE); - msg.m_databaseid = rel->rd_rel->relisshared ? InvalidOid : MyDatabaseId; - msg.m_tableoid = RelationGetRelid(rel); - msg.m_autovacuum = IsAutoVacuumWorkerProcess(); - msg.m_resetcounter = resetcounter; - msg.m_analyzetime = GetCurrentTimestamp(); - msg.m_live_tuples = livetuples; - msg.m_dead_tuples = deadtuples; - pgstat_send(&msg, sizeof(msg)); + /* block acquiring lock for the same reason as pgstat_report_autovac() */ + entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION, dboid, + RelationGetRelid(rel), + false); + /* can't get dropped while accessed */ + Assert(entry_ref != NULL && entry_ref->shared_stats != NULL); + + shtabentry = (PgStatShared_Relation *) entry_ref->shared_stats; + tabentry = &shtabentry->stats; + + tabentry->n_live_tuples = livetuples; + tabentry->n_dead_tuples = deadtuples; + + /* + * If commanded, reset changes_since_analyze to zero. This forgets any + * changes that were committed while the ANALYZE was in progress, but we + * have no good way to estimate how many of those there were. + */ + if (resetcounter) + tabentry->changes_since_analyze = 0; + + if (IsAutoVacuumWorkerProcess()) + { + tabentry->autovac_analyze_timestamp = GetCurrentTimestamp(); + tabentry->autovac_analyze_count++; + } + else + { + tabentry->analyze_timestamp = GetCurrentTimestamp(); + tabentry->analyze_count++; + } + + pgstat_unlock_entry(entry_ref); } /* @@ -376,30 +458,61 @@ pgstat_update_heap_dead_tuples(Relation rel, int delta) } } +/* + * Support function for the SQL-callable pgstat* functions. Returns + * the collected statistics for one table or NULL. NULL doesn't mean + * that the table doesn't exist, just that there are no statistics, so the + * caller is better off to report ZERO instead. + */ +PgStat_StatTabEntry * +pgstat_fetch_stat_tabentry(Oid relid) +{ + PgStat_StatTabEntry *tabentry; + + tabentry = pgstat_fetch_stat_tabentry_ext(false, relid); + if (tabentry != NULL) + return tabentry; + + /* + * If we didn't find it, maybe it's a shared table. + */ + tabentry = pgstat_fetch_stat_tabentry_ext(true, relid); + return tabentry; +} + +/* + * More efficient version of pgstat_fetch_stat_tabentry(), allowing to specify + * whether the to-be-accessed table is a shared relation or not. + */ +PgStat_StatTabEntry * +pgstat_fetch_stat_tabentry_ext(bool shared, Oid reloid) +{ + Oid dboid = (shared ? InvalidOid : MyDatabaseId); + + return (PgStat_StatTabEntry *) + pgstat_fetch_entry(PGSTAT_KIND_RELATION, dboid, reloid); +} + /* * find any existing PgStat_TableStatus entry for rel * - * If no entry, return NULL, don't create a new one + * Find any existing PgStat_TableStatus entry for rel_id in the current + * database. If not found, try finding from shared tables. * - * Note: if we got an error in the most recent execution of pgstat_report_stat, - * it's possible that an entry exists but there's no hashtable entry for it. - * That's okay, we'll treat this case as "doesn't exist". + * If no entry found, return NULL, don't create a new one */ PgStat_TableStatus * find_tabstat_entry(Oid rel_id) { - TabStatHashEntry *hash_entry; + PgStat_EntryRef *entry_ref; - /* If hashtable doesn't exist, there are no entries at all */ - if (!pgStatTabHash) - return NULL; + entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION, MyDatabaseId, rel_id); + if (!entry_ref) + entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION, InvalidOid, rel_id); - hash_entry = hash_search(pgStatTabHash, &rel_id, HASH_FIND, NULL); - if (!hash_entry) - return NULL; - - /* Note that this step could also return NULL, but that's correct */ - return hash_entry->tsa_entry; + if (entry_ref) + return entry_ref->pending; + return NULL; } /* @@ -556,7 +669,7 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state) for (trans = xact_state->first; trans != NULL; trans = trans->next) { - PgStat_TableStatus *tabstat; + PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY; TwoPhasePgStatRecord record; Assert(trans->nest_level == 1); @@ -614,7 +727,7 @@ pgstat_twophase_postcommit(TransactionId xid, uint16 info, PgStat_TableStatus *pgstat_info; /* Find or create a tabstat entry for the rel */ - pgstat_info = get_tabstat_entry(rec->t_id, rec->t_shared); + pgstat_info = pgstat_prep_relation_pending(rec->t_id, rec->t_shared); /* Same math as in AtEOXact_PgStat, commit case */ pgstat_info->t_counts.t_tuples_inserted += rec->tuples_inserted; @@ -650,7 +763,7 @@ pgstat_twophase_postabort(TransactionId xid, uint16 info, PgStat_TableStatus *pgstat_info; /* Find or create a tabstat entry for the rel */ - pgstat_info = get_tabstat_entry(rec->t_id, rec->t_shared); + pgstat_info = pgstat_prep_relation_pending(rec->t_id, rec->t_shared); /* Same math as in AtEOXact_PgStat, abort case */ if (rec->t_truncdropped) @@ -667,204 +780,117 @@ pgstat_twophase_postabort(TransactionId xid, uint16 info, } /* - * Subroutine for pgstat_report_stat: Send relation statistics + * Flush out pending stats for the entry + * + * If nowait is true, this function returns false if lock could not + * immediately acquired, otherwise true is returned. + * + * Some of the stats are copied to the corresponding pending database stats + * entry when successfully flushing. */ -void -pgstat_send_tabstats(TimestampTz now, bool disconnect) +bool +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) { - /* we assume this inits to all zeroes: */ static const PgStat_TableCounts all_zeroes; - PgStat_MsgTabstat regular_msg; - PgStat_MsgTabstat shared_msg; - TabStatusArray *tsa; - int i; + Oid dboid; + PgStat_TableStatus *lstats; /* pending stats entry */ + PgStatShared_Relation *shtabstats; + PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ + PgStat_StatDBEntry *dbentry; /* pending database entry */ + + dboid = entry_ref->shared_entry->key.dboid; + lstats = (PgStat_TableStatus *) entry_ref->pending; + shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; /* - * Destroy pgStatTabHash before we start invalidating PgStat_TableEntry - * entries it points to. (Should we fail partway through the loop below, - * it's okay to have removed the hashtable already --- the only - * consequence is we'd get multiple entries for the same table in the - * pgStatTabList, and that's safe.) + * Ignore entries that didn't accumulate any actual counts, such as + * indexes that were opened by the planner but not used. */ - if (pgStatTabHash) - hash_destroy(pgStatTabHash); - pgStatTabHash = NULL; - - /* - * Scan through the TabStatusArray struct(s) to find tables that actually - * have counts, and build messages to send. We have to separate shared - * relations from regular ones because the databaseid field in the message - * header has to depend on that. - */ - regular_msg.m_databaseid = MyDatabaseId; - shared_msg.m_databaseid = InvalidOid; - regular_msg.m_nentries = 0; - shared_msg.m_nentries = 0; - - for (tsa = pgStatTabList; tsa != NULL; tsa = tsa->tsa_next) + if (memcmp(&lstats->t_counts, &all_zeroes, + sizeof(PgStat_TableCounts)) == 0) { - for (i = 0; i < tsa->tsa_used; i++) - { - PgStat_TableStatus *entry = &tsa->tsa_entries[i]; - PgStat_MsgTabstat *this_msg; - PgStat_TableEntry *this_ent; - - /* Shouldn't have any pending transaction-dependent counts */ - Assert(entry->trans == NULL); - - /* - * Ignore entries that didn't accumulate any actual counts, such - * as indexes that were opened by the planner but not used. - */ - if (memcmp(&entry->t_counts, &all_zeroes, - sizeof(PgStat_TableCounts)) == 0) - continue; - - /* - * OK, insert data into the appropriate message, and send if full. - */ - this_msg = entry->t_shared ? &shared_msg : ®ular_msg; - this_ent = &this_msg->m_entry[this_msg->m_nentries]; - this_ent->t_id = entry->t_id; - memcpy(&this_ent->t_counts, &entry->t_counts, - sizeof(PgStat_TableCounts)); - if (++this_msg->m_nentries >= PGSTAT_NUM_TABENTRIES) - { - pgstat_send_tabstat(this_msg, now); - this_msg->m_nentries = 0; - } - } - /* zero out PgStat_TableStatus structs after use */ - MemSet(tsa->tsa_entries, 0, - tsa->tsa_used * sizeof(PgStat_TableStatus)); - tsa->tsa_used = 0; + return true; } - /* - * Send partial messages. Make sure that any pending xact commit/abort - * and connection stats get counted, even if there are no table stats to - * send. - */ - if (regular_msg.m_nentries > 0 || - pgStatXactCommit > 0 || pgStatXactRollback > 0 || disconnect) - pgstat_send_tabstat(®ular_msg, now); - if (shared_msg.m_nentries > 0) - pgstat_send_tabstat(&shared_msg, now); + if (!pgstat_lock_entry(entry_ref, nowait)) + return false; - have_relation_stats = false; + /* add the values to the shared entry. */ + tabentry = &shtabstats->stats; + + tabentry->numscans += lstats->t_counts.t_numscans; + tabentry->tuples_returned += lstats->t_counts.t_tuples_returned; + tabentry->tuples_fetched += lstats->t_counts.t_tuples_fetched; + tabentry->tuples_inserted += lstats->t_counts.t_tuples_inserted; + tabentry->tuples_updated += lstats->t_counts.t_tuples_updated; + tabentry->tuples_deleted += lstats->t_counts.t_tuples_deleted; + tabentry->tuples_hot_updated += lstats->t_counts.t_tuples_hot_updated; + + /* + * If table was truncated or vacuum/analyze has ran, first reset the + * live/dead counters. + */ + if (lstats->t_counts.t_truncdropped) + { + tabentry->n_live_tuples = 0; + tabentry->n_dead_tuples = 0; + tabentry->inserts_since_vacuum = 0; + } + + tabentry->n_live_tuples += lstats->t_counts.t_delta_live_tuples; + tabentry->n_dead_tuples += lstats->t_counts.t_delta_dead_tuples; + tabentry->changes_since_analyze += lstats->t_counts.t_changed_tuples; + tabentry->inserts_since_vacuum += lstats->t_counts.t_tuples_inserted; + tabentry->blocks_fetched += lstats->t_counts.t_blocks_fetched; + tabentry->blocks_hit += lstats->t_counts.t_blocks_hit; + + /* Clamp n_live_tuples in case of negative delta_live_tuples */ + tabentry->n_live_tuples = Max(tabentry->n_live_tuples, 0); + /* Likewise for n_dead_tuples */ + tabentry->n_dead_tuples = Max(tabentry->n_dead_tuples, 0); + + pgstat_unlock_entry(entry_ref); + + /* The entry is successfully flushed so the same to add to database stats */ + dbentry = pgstat_prep_database_pending(dboid); + dbentry->n_tuples_returned += lstats->t_counts.t_tuples_returned; + dbentry->n_tuples_fetched += lstats->t_counts.t_tuples_fetched; + dbentry->n_tuples_inserted += lstats->t_counts.t_tuples_inserted; + dbentry->n_tuples_updated += lstats->t_counts.t_tuples_updated; + dbentry->n_tuples_deleted += lstats->t_counts.t_tuples_deleted; + dbentry->n_blocks_fetched += lstats->t_counts.t_blocks_fetched; + dbentry->n_blocks_hit += lstats->t_counts.t_blocks_hit; + + return true; } -/* - * Subroutine for pgstat_send_tabstats: finish and send one tabstat message - */ -static void -pgstat_send_tabstat(PgStat_MsgTabstat *tsmsg, TimestampTz now) +void +pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref) { - int n; - int len; + PgStat_TableStatus *pending = (PgStat_TableStatus *) entry_ref->pending; - /* It's unlikely we'd get here with no socket, but maybe not impossible */ - if (pgStatSock == PGINVALID_SOCKET) - return; - - /* - * Report and reset accumulated xact commit/rollback and I/O timings - * whenever we send a normal tabstat message - */ - pgstat_update_dbstats(tsmsg, now); - - n = tsmsg->m_nentries; - len = offsetof(PgStat_MsgTabstat, m_entry[0]) + - n * sizeof(PgStat_TableEntry); - - pgstat_setheader(&tsmsg->m_hdr, PGSTAT_MTYPE_TABSTAT); - pgstat_send(tsmsg, len); + if (pending->relation) + pgstat_relation_unlink(pending->relation); } /* - * find or create a PgStat_TableStatus entry for rel + * Find or create a PgStat_TableStatus entry for rel. New entry is created and + * initialized if not exists. */ static PgStat_TableStatus * -get_tabstat_entry(Oid rel_id, bool isshared) +pgstat_prep_relation_pending(Oid rel_id, bool isshared) { - TabStatHashEntry *hash_entry; - PgStat_TableStatus *entry; - TabStatusArray *tsa; - bool found; + PgStat_EntryRef *entry_ref; + PgStat_TableStatus *pending; - pgstat_assert_is_up(); + entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_RELATION, + isshared ? InvalidOid : MyDatabaseId, + rel_id, NULL); + pending = entry_ref->pending; + pending->t_id = rel_id; + pending->t_shared = isshared; - have_relation_stats = true; - - /* - * Create hash table if we don't have it already. - */ - if (pgStatTabHash == NULL) - { - HASHCTL ctl; - - ctl.keysize = sizeof(Oid); - ctl.entrysize = sizeof(TabStatHashEntry); - - pgStatTabHash = hash_create("pgstat TabStatusArray lookup hash table", - TABSTAT_QUANTUM, - &ctl, - HASH_ELEM | HASH_BLOBS); - } - - /* - * Find an entry or create a new one. - */ - hash_entry = hash_search(pgStatTabHash, &rel_id, HASH_ENTER, &found); - if (!found) - { - /* initialize new entry with null pointer */ - hash_entry->tsa_entry = NULL; - } - - /* - * If entry is already valid, we're done. - */ - if (hash_entry->tsa_entry) - return hash_entry->tsa_entry; - - /* - * Locate the first pgStatTabList entry with free space, making a new list - * entry if needed. Note that we could get an OOM failure here, but if so - * we have left the hashtable and the list in a consistent state. - */ - if (pgStatTabList == NULL) - { - /* Set up first pgStatTabList entry */ - pgStatTabList = (TabStatusArray *) - MemoryContextAllocZero(TopMemoryContext, - sizeof(TabStatusArray)); - } - - tsa = pgStatTabList; - while (tsa->tsa_used >= TABSTAT_QUANTUM) - { - if (tsa->tsa_next == NULL) - tsa->tsa_next = (TabStatusArray *) - MemoryContextAllocZero(TopMemoryContext, - sizeof(TabStatusArray)); - tsa = tsa->tsa_next; - } - - /* - * Allocate a PgStat_TableStatus entry within this list entry. We assume - * the entry was already zeroed, either at creation or after last use. - */ - entry = &tsa->tsa_entries[tsa->tsa_used++]; - entry->t_id = rel_id; - entry->t_shared = isshared; - - /* - * Now we can fill the entry in pgStatTabHash. - */ - hash_entry->tsa_entry = entry; - - return entry; + return pending; } /* diff --git a/src/backend/utils/activity/pgstat_replslot.c b/src/backend/utils/activity/pgstat_replslot.c index b1e894f2b21..c8c2aaadaa5 100644 --- a/src/backend/utils/activity/pgstat_replslot.c +++ b/src/backend/utils/activity/pgstat_replslot.c @@ -19,23 +19,20 @@ #include "replication/slot.h" #include "utils/builtins.h" /* for namestrcpy() */ +#include "utils/memutils.h" #include "utils/pgstat_internal.h" +static int replslot_index(const char *name); + + /* * Reset counters for all replication slots. */ void pgstat_reset_replslot_counters(void) { - PgStat_MsgResetreplslotcounter msg; - - if (pgStatSock == PGINVALID_SOCKET) - return; - - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETREPLSLOTCOUNTER); - msg.clearall = true; - pgstat_send(&msg, sizeof(msg)); + pgstat_reset_entries_of_kind(PGSTAT_KIND_REPLSLOT); } /* @@ -48,18 +45,10 @@ void pgstat_reset_replslot_counter(const char *name) { ReplicationSlot *slot; - PgStat_MsgResetreplslotcounter msg; AssertArg(name != NULL); - if (pgStatSock == PGINVALID_SOCKET) - return; - - /* - * Check if the slot exists with the given name. It is possible that by - * the time this message is executed the slot is dropped but at least this - * check will ensure that the given name is for a valid slot. - */ + /* Check if the slot exits with the given name. */ slot = SearchNamedReplicationSlot(name, true); if (!slot) @@ -75,10 +64,9 @@ pgstat_reset_replslot_counter(const char *name) if (SlotIsPhysical(slot)) return; - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETREPLSLOTCOUNTER); - namestrcpy(&msg.m_slotname, name); - msg.clearall = false; - pgstat_send(&msg, sizeof(msg)); + /* reset this one entry */ + pgstat_reset_entry(PGSTAT_KIND_REPLSLOT, InvalidOid, + ReplicationSlotIndex(slot)); } /* @@ -87,24 +75,34 @@ pgstat_reset_replslot_counter(const char *name) void pgstat_report_replslot(ReplicationSlot *slot, const PgStat_StatReplSlotEntry *repSlotStat) { - PgStat_MsgReplSlot msg; + PgStat_EntryRef *entry_ref; + PgStatShared_ReplSlot *shstatent; + PgStat_StatReplSlotEntry *statent; + + entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_REPLSLOT, InvalidOid, + ReplicationSlotIndex(slot), false); + shstatent = (PgStatShared_ReplSlot *) entry_ref->shared_stats; + statent = &shstatent->stats; /* - * Prepare and send the message + * Any mismatch should have been fixed in pgstat_create_replslot() or + * pgstat_acquire_replslot(). */ - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_REPLSLOT); - namestrcpy(&msg.m_slotname, NameStr(repSlotStat->slotname)); - msg.m_create = false; - msg.m_drop = false; - msg.m_spill_txns = repSlotStat->spill_txns; - msg.m_spill_count = repSlotStat->spill_count; - msg.m_spill_bytes = repSlotStat->spill_bytes; - msg.m_stream_txns = repSlotStat->stream_txns; - msg.m_stream_count = repSlotStat->stream_count; - msg.m_stream_bytes = repSlotStat->stream_bytes; - msg.m_total_txns = repSlotStat->total_txns; - msg.m_total_bytes = repSlotStat->total_bytes; - pgstat_send(&msg, sizeof(PgStat_MsgReplSlot)); + Assert(namestrcmp(&statent->slotname, NameStr(slot->data.name)) == 0); + + /* Update the replication slot statistics */ +#define REPLSLOT_ACC(fld) statent->fld += repSlotStat->fld + REPLSLOT_ACC(spill_txns); + REPLSLOT_ACC(spill_count); + REPLSLOT_ACC(spill_bytes); + REPLSLOT_ACC(stream_txns); + REPLSLOT_ACC(stream_count); + REPLSLOT_ACC(stream_bytes); + REPLSLOT_ACC(total_txns); + REPLSLOT_ACC(total_bytes); +#undef REPLSLOT_ACC + + pgstat_unlock_entry(entry_ref); } /* @@ -116,26 +114,117 @@ pgstat_report_replslot(ReplicationSlot *slot, const PgStat_StatReplSlotEntry *re void pgstat_create_replslot(ReplicationSlot *slot) { - PgStat_MsgReplSlot msg; + PgStat_EntryRef *entry_ref; + PgStatShared_ReplSlot *shstatent; - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_REPLSLOT); - namestrcpy(&msg.m_slotname, NameStr(slot->data.name)); - msg.m_create = true; - msg.m_drop = false; - pgstat_send(&msg, sizeof(PgStat_MsgReplSlot)); + entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_REPLSLOT, InvalidOid, + ReplicationSlotIndex(slot), false); + shstatent = (PgStatShared_ReplSlot *) entry_ref->shared_stats; + + /* + * NB: need to accept that there might be stats from an older slot, e.g. + * if we previously crashed after dropping a slot. + */ + memset(&shstatent->stats, 0, sizeof(shstatent->stats)); + namestrcpy(&shstatent->stats.slotname, NameStr(slot->data.name)); + + pgstat_unlock_entry(entry_ref); +} + +/* + * Report replication slot has been acquired. + */ +void +pgstat_acquire_replslot(ReplicationSlot *slot) +{ + PgStat_EntryRef *entry_ref; + PgStatShared_ReplSlot *shstatent; + PgStat_StatReplSlotEntry *statent; + + entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_REPLSLOT, InvalidOid, + ReplicationSlotIndex(slot), false); + shstatent = (PgStatShared_ReplSlot *) entry_ref->shared_stats; + statent = &shstatent->stats; + + /* + * NB: need to accept that there might be stats from an older slot, e.g. + * if we previously crashed after dropping a slot. + */ + if (NameStr(statent->slotname)[0] == 0 || + namestrcmp(&statent->slotname, NameStr(slot->data.name)) != 0) + { + memset(statent, 0, sizeof(*statent)); + namestrcpy(&statent->slotname, NameStr(slot->data.name)); + } + + pgstat_unlock_entry(entry_ref); } /* * Report replication slot drop. */ void -pgstat_drop_replslot(const char *slotname) +pgstat_drop_replslot(ReplicationSlot *slot) { - PgStat_MsgReplSlot msg; - - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_REPLSLOT); - namestrcpy(&msg.m_slotname, slotname); - msg.m_create = false; - msg.m_drop = true; - pgstat_send(&msg, sizeof(PgStat_MsgReplSlot)); + pgstat_drop_entry(PGSTAT_KIND_REPLSLOT, InvalidOid, + ReplicationSlotIndex(slot)); +} + +/* + * Support function for the SQL-callable pgstat* functions. Returns + * a pointer to the replication slot statistics struct. + */ +PgStat_StatReplSlotEntry * +pgstat_fetch_replslot(NameData slotname) +{ + int idx = replslot_index(NameStr(slotname)); + + if (idx == -1) + return NULL; + + return (PgStat_StatReplSlotEntry *) + pgstat_fetch_entry(PGSTAT_KIND_REPLSLOT, InvalidOid, idx); +} + +void +pgstat_replslot_to_serialized_name_cb(const PgStatShared_Common *header, NameData *name) +{ + namestrcpy(name, NameStr(((PgStatShared_ReplSlot *) header)->stats.slotname)); +} + +bool +pgstat_replslot_from_serialized_name_cb(const NameData *name, PgStat_HashKey *key) +{ + int idx = replslot_index(NameStr(*name)); + + /* slot might have been deleted */ + if (idx == -1) + return false; + + key->kind = PGSTAT_KIND_REPLSLOT; + key->dboid = InvalidOid; + key->objoid = idx; + + return true; +} + +void +pgstat_replslot_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts) +{ + ((PgStatShared_ReplSlot *) header)->stats.stat_reset_timestamp = ts; +} + +static int +replslot_index(const char *name) +{ + ReplicationSlot *slot; + + AssertArg(name != NULL); + + slot = SearchNamedReplicationSlot(name, true); + + if (!slot) + return -1; + + return ReplicationSlotIndex(slot); } diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c new file mode 100644 index 00000000000..d11b628710f --- /dev/null +++ b/src/backend/utils/activity/pgstat_shmem.c @@ -0,0 +1,998 @@ +/* ------------------------------------------------------------------------- + * + * pgstat_shmem.c + * Storage of stats entries in shared memory + * + * Copyright (c) 2001-2022, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/utils/activity/pgstat_shmem.c + * ------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/transam.h" +#include "access/xact.h" +#include "pgstat.h" +#include "storage/shmem.h" +#include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timestamp.h" +#include "common/hashfn.h" + + +#define PGSTAT_ENTRY_REF_HASH_SIZE 128 + +/* hash table entry for finding the PgStat_EntryRef for a key */ +typedef struct PgStat_EntryRefHashEntry +{ + PgStat_HashKey key; /* hash key */ + char status; /* for simplehash use */ + PgStat_EntryRef *entry_ref; +} PgStat_EntryRefHashEntry; + + +/* for references to shared statistics entries */ +#define SH_PREFIX pgstat_entry_ref_hash +#define SH_ELEMENT_TYPE PgStat_EntryRefHashEntry +#define SH_KEY_TYPE PgStat_HashKey +#define SH_KEY key +#define SH_HASH_KEY(tb, key) \ + hash_bytes((unsigned char *)&key, sizeof(PgStat_HashKey)) +#define SH_EQUAL(tb, a, b) (memcmp(&a, &b, sizeof(PgStat_HashKey)) == 0) +#define SH_SCOPE static inline +#define SH_DEFINE +#define SH_DECLARE +#include "lib/simplehash.h" + + +static void pgstat_drop_database_and_contents(Oid dboid); + +static void pgstat_free_entry(PgStatShared_HashEntry *shent, dshash_seq_status *hstat); + +static void pgstat_release_entry_ref(PgStat_HashKey key, PgStat_EntryRef *entry_ref, bool discard_pending); +static bool pgstat_need_entry_refs_gc(void); +static void pgstat_gc_entry_refs(void); +static void pgstat_release_all_entry_refs(bool discard_pending); +typedef bool (*ReleaseMatchCB) (PgStat_EntryRefHashEntry *, Datum data); +static void pgstat_release_matching_entry_refs(bool discard_pending, ReleaseMatchCB match, Datum match_data); + + +static void pgstat_setup_memcxt(void); + + +/* parameter for the shared hash */ +static const dshash_parameters dsh_params = { + sizeof(PgStat_HashKey), + sizeof(PgStatShared_HashEntry), + dshash_memcmp, + dshash_memhash, + LWTRANCHE_PGSTATS_HASH +}; + + +/* + * Backend local references to shared stats entries. If there are pending + * updates to a stats entry, the PgStat_EntryRef is added to the pgStatPending + * list. + * + * When a stats entry is dropped each backend needs to release its reference + * to it before the memory can be released. To trigger that + * pgStatLocal.shmem->gc_count is incremented - which each backend compares to + * their copy of pgStatSharedRefAge on a regular basis. + */ +static pgstat_entry_ref_hash_hash *pgStatEntryRefHash = NULL; +static int pgStatSharedRefAge = 0; /* cache age of pgStatShmLookupCache */ + +/* + * Memory contexts containing the pgStatEntryRefHash table and the + * pgStatSharedRef entries respectively. Kept separate to make it easier to + * track / attribute memory usage. + */ +static MemoryContext pgStatSharedRefContext = NULL; +static MemoryContext pgStatEntryRefHashContext = NULL; + + +/* ------------------------------------------------------------ + * Public functions called from postmaster follow + * ------------------------------------------------------------ + */ + +/* + * The size of the shared memory allocation for stats stored in the shared + * stats hash table. This allocation will be done as part of the main shared + * memory, rather than dynamic shared memory, allowing it to be initialized in + * postmaster. + */ +static Size +pgstat_dsa_init_size(void) +{ + Size sz; + + /* + * The dshash header / initial buckets array needs to fit into "plain" + * shared memory, but it's beneficial to not need dsm segments + * immediately. A size of 256kB seems works well and is not + * disproportional compared to other constant sized shared memory + * allocations. NB: To avoid DSMs further, the user can configure + * min_dynamic_shared_memory. + */ + sz = 256 * 1024; + Assert(dsa_minimum_size() <= sz); + return MAXALIGN(sz); +} + +/* + * Compute shared memory space needed for cumulative statistics + */ +Size +StatsShmemSize(void) +{ + Size sz; + + sz = MAXALIGN(sizeof(PgStat_ShmemControl)); + sz = add_size(sz, pgstat_dsa_init_size()); + + return sz; +} + +/* + * Initialize cumulative statistics system during startup + */ +void +StatsShmemInit(void) +{ + bool found; + Size sz; + + sz = StatsShmemSize(); + pgStatLocal.shmem = (PgStat_ShmemControl *) + ShmemInitStruct("Shared Memory Stats", sz, &found); + + if (!IsUnderPostmaster) + { + dsa_area *dsa; + dshash_table *dsh; + char *p = (char *) pgStatLocal.shmem; + + Assert(!found); + + /* the allocation of pgStatLocal.shmem itself */ + p += MAXALIGN(sizeof(PgStat_ShmemControl)); + + /* + * Create a small dsa allocation in plain shared memory. Doing so + * initially makes it easier to manage server startup, and it also is + * a small efficiency win. + */ + pgStatLocal.shmem->raw_dsa_area = p; + p += MAXALIGN(pgstat_dsa_init_size()); + dsa = dsa_create_in_place(pgStatLocal.shmem->raw_dsa_area, + pgstat_dsa_init_size(), + LWTRANCHE_PGSTATS_DSA, 0); + dsa_pin(dsa); + + /* + * To ensure dshash is created in "plain" shared memory, temporarily + * limit size of dsa to the initial size of the dsa. + */ + dsa_set_size_limit(dsa, pgstat_dsa_init_size()); + + /* + * With the limit in place, create the dshash table. XXX: It'd be nice + * if there were dshash_create_in_place(). + */ + dsh = dshash_create(dsa, &dsh_params, 0); + pgStatLocal.shmem->hash_handle = dshash_get_hash_table_handle(dsh); + + /* lift limit set above */ + dsa_set_size_limit(dsa, -1); + + /* + * Postmaster will never access these again, thus free the local + * dsa/dshash references. + */ + dshash_detach(dsh); + dsa_detach(dsa); + + pg_atomic_init_u64(&pgStatLocal.shmem->gc_count, 1); + + + /* + * Initialize global statistics. + */ + + LWLockInitialize(&pgStatLocal.shmem->archiver.lock, LWTRANCHE_PGSTATS_DATA); + LWLockInitialize(&pgStatLocal.shmem->bgwriter.lock, LWTRANCHE_PGSTATS_DATA); + LWLockInitialize(&pgStatLocal.shmem->checkpointer.lock, LWTRANCHE_PGSTATS_DATA); + LWLockInitialize(&pgStatLocal.shmem->slru.lock, LWTRANCHE_PGSTATS_DATA); + LWLockInitialize(&pgStatLocal.shmem->wal.lock, LWTRANCHE_PGSTATS_DATA); + } + else + { + Assert(found); + } +} + +void +pgstat_attach_shmem(void) +{ + MemoryContext oldcontext; + + Assert(pgStatLocal.dsa == NULL); + + /* stats shared memory persists for the backend lifetime */ + oldcontext = MemoryContextSwitchTo(TopMemoryContext); + + pgStatLocal.dsa = dsa_attach_in_place(pgStatLocal.shmem->raw_dsa_area, NULL); + dsa_pin_mapping(pgStatLocal.dsa); + + pgStatLocal.shared_hash = dshash_attach(pgStatLocal.dsa, &dsh_params, + pgStatLocal.shmem->hash_handle, 0); + + MemoryContextSwitchTo(oldcontext); +} + +void +pgstat_detach_shmem(void) +{ + Assert(pgStatLocal.dsa); + + /* we shouldn't leave references to shared stats */ + pgstat_release_all_entry_refs(false); + + dshash_detach(pgStatLocal.shared_hash); + pgStatLocal.shared_hash = NULL; + + dsa_detach(pgStatLocal.dsa); + pgStatLocal.dsa = NULL; +} + + +/* ------------------------------------------------------------ + * Maintenance of shared memory stats entries + * ------------------------------------------------------------ + */ + +PgStatShared_Common * +pgstat_init_entry(PgStat_Kind kind, + PgStatShared_HashEntry *shhashent) +{ + /* Create new stats entry. */ + dsa_pointer chunk; + PgStatShared_Common *shheader; + + /* + * Initialize refcount to 1, marking it as valid / not tdroped. The entry + * can't be freed before the initialization because it can't be found as + * long as we hold the dshash partition lock. Caller needs to increase + * further if a longer lived references is needed. + */ + pg_atomic_init_u32(&shhashent->refcount, 1); + shhashent->dropped = false; + + chunk = dsa_allocate0(pgStatLocal.dsa, pgstat_kind_info_for(kind)->shared_size); + shheader = dsa_get_address(pgStatLocal.dsa, chunk); + shheader->magic = 0xdeadbeef; + + /* Link the new entry from the hash entry. */ + shhashent->body = chunk; + + LWLockInitialize(&shheader->lock, LWTRANCHE_PGSTATS_DATA); + + return shheader; +} + +/* + * There are legitimate cases where the old stats entry might not yet + * have been dropped by the time it's reused. The easiest case are + * replication slot stats. But oid wraparound can lead to other cases + * as well. We just reset the stats to their plain state. + */ +static PgStatShared_Common * +pgstat_reinit_entry(PgStat_Kind kind, PgStatShared_HashEntry *shhashent) +{ + PgStatShared_Common *shheader; + + shheader = dsa_get_address(pgStatLocal.dsa, shhashent->body); + + /* mark as not dropped anymore */ + pg_atomic_fetch_add_u32(&shhashent->refcount, 1); + shhashent->dropped = false; + + /* reinitialize content */ + Assert(shheader->magic == 0xdeadbeef); + memset(shheader, 0, pgstat_kind_info_for(shhashent->key.kind)->shared_size); + shheader->magic = 0xdeadbeef; + + return shheader; +} + +static void +pgstat_setup_shared_refs(void) +{ + if (likely(pgStatEntryRefHash != NULL)) + return; + + pgStatEntryRefHash = + pgstat_entry_ref_hash_create(pgStatEntryRefHashContext, + PGSTAT_ENTRY_REF_HASH_SIZE, NULL); + pgStatSharedRefAge = pg_atomic_read_u64(&pgStatLocal.shmem->gc_count); + Assert(pgStatSharedRefAge != 0); +} + +/* + * Helper function for pgstat_get_entry_ref(). + */ +static void +pgstat_acquire_entry_ref(PgStat_EntryRef *entry_ref, + PgStatShared_HashEntry *shhashent, + PgStatShared_Common *shheader) +{ + Assert(shheader->magic == 0xdeadbeef); + Assert(pg_atomic_read_u32(&shhashent->refcount) > 0); + + pg_atomic_fetch_add_u32(&shhashent->refcount, 1); + + dshash_release_lock(pgStatLocal.shared_hash, shhashent); + + entry_ref->shared_stats = shheader; + entry_ref->shared_entry = shhashent; +} + +/* + * Helper function for pgstat_get_entry_ref(). + */ +static bool +pgstat_get_entry_ref_cached(PgStat_HashKey key, PgStat_EntryRef **entry_ref_p) +{ + bool found; + PgStat_EntryRefHashEntry *cache_entry; + + /* + * We immediately insert a cache entry, because it avoids 1) multiple + * hashtable lookups in case of a cache miss 2) having to deal with + * out-of-memory errors after incrementing PgStatShared_Common->refcount. + */ + + cache_entry = pgstat_entry_ref_hash_insert(pgStatEntryRefHash, key, &found); + + if (!found || !cache_entry->entry_ref) + { + PgStat_EntryRef *entry_ref; + + cache_entry->entry_ref = entry_ref = + MemoryContextAlloc(pgStatSharedRefContext, + sizeof(PgStat_EntryRef)); + entry_ref->shared_stats = NULL; + entry_ref->shared_entry = NULL; + entry_ref->pending = NULL; + + found = false; + } + else if (cache_entry->entry_ref->shared_stats == NULL) + { + Assert(cache_entry->entry_ref->pending == NULL); + found = false; + } + else + { + PgStat_EntryRef *entry_ref PG_USED_FOR_ASSERTS_ONLY; + + entry_ref = cache_entry->entry_ref; + Assert(entry_ref->shared_entry != NULL); + Assert(entry_ref->shared_stats != NULL); + + Assert(entry_ref->shared_stats->magic == 0xdeadbeef); + /* should have at least our reference */ + Assert(pg_atomic_read_u32(&entry_ref->shared_entry->refcount) > 0); + } + + *entry_ref_p = cache_entry->entry_ref; + return found; +} + +/* + * Get a shared stats reference. If create is true, the shared stats object is + * created if it does not exist. + * + * When create is true, and created_entry is non-NULL, it'll be set to true + * if the entry is newly created, false otherwise. + */ +PgStat_EntryRef * +pgstat_get_entry_ref(PgStat_Kind kind, Oid dboid, Oid objoid, bool create, + bool *created_entry) +{ + PgStat_HashKey key = {.kind = kind,.dboid = dboid,.objoid = objoid}; + PgStatShared_HashEntry *shhashent; + PgStatShared_Common *shheader = NULL; + PgStat_EntryRef *entry_ref; + + /* + * passing in created_entry only makes sense if we possibly could create + * entry. + */ + AssertArg(create || created_entry == NULL); + pgstat_assert_is_up(); + Assert(pgStatLocal.shared_hash != NULL); + Assert(!pgStatLocal.shmem->is_shutdown); + + pgstat_setup_memcxt(); + pgstat_setup_shared_refs(); + + if (created_entry != NULL) + *created_entry = false; + + /* + * Check if other backends dropped stats that could not be deleted because + * somebody held references to it. If so, check this backend's references. + * This is not expected to happen often. The location of the check is a + * bit random, but this is a relatively frequently called path, so better + * than most. + */ + if (pgstat_need_entry_refs_gc()) + pgstat_gc_entry_refs(); + + /* + * First check the lookup cache hashtable in local memory. If we find a + * match here we can avoid taking locks / causing contention. + */ + if (pgstat_get_entry_ref_cached(key, &entry_ref)) + return entry_ref; + + Assert(entry_ref != NULL); + + /* + * Do a lookup in the hash table first - it's quite likely that the entry + * already exists, and that way we only need a shared lock. + */ + shhashent = dshash_find(pgStatLocal.shared_hash, &key, false); + + if (create && !shhashent) + { + bool shfound; + + /* + * It's possible that somebody created the entry since the above + * lookup. If so, fall through to the same path as if we'd have if it + * already had been created before the dshash_find() calls. + */ + shhashent = dshash_find_or_insert(pgStatLocal.shared_hash, &key, &shfound); + if (!shfound) + { + shheader = pgstat_init_entry(kind, shhashent); + pgstat_acquire_entry_ref(entry_ref, shhashent, shheader); + + if (created_entry != NULL) + *created_entry = true; + + return entry_ref; + } + } + + if (!shhashent) + { + /* + * If we're not creating, delete the reference again. In all + * likelihood it's just a stats lookup - no point wasting memory for a + * shared ref to nothing... + */ + pgstat_release_entry_ref(key, entry_ref, false); + + return NULL; + } + else + { + /* + * Can get here either because dshash_find() found a match, or if + * dshash_find_or_insert() found a concurrently inserted entry. + */ + + if (shhashent->dropped && create) + { + /* + * There are legitimate cases where the old stats entry might not + * yet have been dropped by the time it's reused. The easiest case + * are replication slot stats. But oid wraparound can lead to + * other cases as well. We just reset the stats to their plain + * state. + */ + shheader = pgstat_reinit_entry(kind, shhashent); + pgstat_acquire_entry_ref(entry_ref, shhashent, shheader); + + if (created_entry != NULL) + *created_entry = true; + + return entry_ref; + } + else if (shhashent->dropped) + { + dshash_release_lock(pgStatLocal.shared_hash, shhashent); + pgstat_release_entry_ref(key, entry_ref, false); + + return NULL; + } + else + { + shheader = dsa_get_address(pgStatLocal.dsa, shhashent->body); + pgstat_acquire_entry_ref(entry_ref, shhashent, shheader); + + return entry_ref; + } + } +} + +static void +pgstat_release_entry_ref(PgStat_HashKey key, PgStat_EntryRef *entry_ref, bool discard_pending) +{ + if (entry_ref && entry_ref->pending) + { + if (discard_pending) + pgstat_delete_pending_entry(entry_ref); + else + elog(ERROR, "releasing ref with pending data"); + } + + if (entry_ref && entry_ref->shared_stats) + { + Assert(entry_ref->shared_stats->magic == 0xdeadbeef); + Assert(entry_ref->pending == NULL); + + /* + * This can't race with another backend looking up the stats entry and + * increasing the refcount because it is not "legal" to create + * additional references to dropped entries. + */ + if (pg_atomic_fetch_sub_u32(&entry_ref->shared_entry->refcount, 1) == 1) + { + PgStatShared_HashEntry *shent; + + /* + * We're the last referrer to this entry, try to drop the shared + * entry. + */ + + /* only dropped entries can reach a 0 refcount */ + Assert(entry_ref->shared_entry->dropped); + + shent = dshash_find(pgStatLocal.shared_hash, + &entry_ref->shared_entry->key, + true); + if (!shent) + elog(PANIC, "could not find just referenced shared stats entry"); + + /* ATODO: convert to assertion before commit */ + if (pg_atomic_read_u32(&entry_ref->shared_entry->refcount) != 0) + elog(PANIC, "concurrent access to stats entry during deletion"); + + Assert(entry_ref->shared_entry == shent); + + pgstat_free_entry(shent, NULL); + } + } + + if (!pgstat_entry_ref_hash_delete(pgStatEntryRefHash, key)) + elog(PANIC, "something has gone wrong"); + + if (entry_ref) + pfree(entry_ref); +} + +bool +pgstat_lock_entry(PgStat_EntryRef *entry_ref, bool nowait) +{ + LWLock *lock = &entry_ref->shared_stats->lock; + + if (nowait) + return LWLockConditionalAcquire(lock, LW_EXCLUSIVE); + + LWLockAcquire(lock, LW_EXCLUSIVE); + return true; +} + +void +pgstat_unlock_entry(PgStat_EntryRef *entry_ref) +{ + LWLockRelease(&entry_ref->shared_stats->lock); +} + +/* + * Helper function to fetch and lock shared stats. + */ +PgStat_EntryRef * +pgstat_get_entry_ref_locked(PgStat_Kind kind, Oid dboid, Oid objoid, bool nowait) +{ + PgStat_EntryRef *entry_ref; + + /* find shared table stats entry corresponding to the local entry */ + entry_ref = pgstat_get_entry_ref(kind, dboid, objoid, true, NULL); + + /* lock the shared entry to protect the content, skip if failed */ + if (!pgstat_lock_entry(entry_ref, nowait)) + return NULL; + + return entry_ref; +} + +void +pgstat_request_entry_refs_gc(void) +{ + pg_atomic_fetch_add_u64(&pgStatLocal.shmem->gc_count, 1); +} + +static bool +pgstat_need_entry_refs_gc(void) +{ + uint64 curage; + + if (!pgStatEntryRefHash) + return false; + + /* should have been initialized when creating pgStatEntryRefHash */ + Assert(pgStatSharedRefAge != 0); + + curage = pg_atomic_read_u64(&pgStatLocal.shmem->gc_count); + + return pgStatSharedRefAge != curage; +} + +static void +pgstat_gc_entry_refs(void) +{ + pgstat_entry_ref_hash_iterator i; + PgStat_EntryRefHashEntry *ent; + uint64 curage; + + curage = pg_atomic_read_u64(&pgStatLocal.shmem->gc_count); + Assert(curage != 0); + + /* + * Some entries have been dropped. Invalidate cache pointer to them. + */ + pgstat_entry_ref_hash_start_iterate(pgStatEntryRefHash, &i); + while ((ent = pgstat_entry_ref_hash_iterate(pgStatEntryRefHash, &i)) != NULL) + { + PgStat_EntryRef *entry_ref = ent->entry_ref; + + Assert(!entry_ref->shared_stats || entry_ref->shared_stats->magic == 0xdeadbeef); + + if (!entry_ref->shared_entry->dropped) + continue; + + /* cannot gc shared ref that has pending data */ + if (entry_ref->pending != NULL) + continue; + + pgstat_release_entry_ref(ent->key, entry_ref, false); + } + + pgStatSharedRefAge = curage; +} + +static void +pgstat_release_matching_entry_refs(bool discard_pending, ReleaseMatchCB match, Datum match_data) +{ + pgstat_entry_ref_hash_iterator i; + PgStat_EntryRefHashEntry *ent; + + if (pgStatEntryRefHash == NULL) + return; + + pgstat_entry_ref_hash_start_iterate(pgStatEntryRefHash, &i); + + while ((ent = pgstat_entry_ref_hash_iterate(pgStatEntryRefHash, &i)) + != NULL) + { + Assert(ent->entry_ref != NULL); + + if (match && !match(ent, match_data)) + continue; + + pgstat_release_entry_ref(ent->key, ent->entry_ref, discard_pending); + } +} + +/* + * Release all local references to shared stats entries. + * + * When a process exits it cannot do so while still holding references onto + * stats entries, otherwise the shared stats entries could never be freed. + */ +static void +pgstat_release_all_entry_refs(bool discard_pending) +{ + if (pgStatEntryRefHash == NULL) + return; + + pgstat_release_matching_entry_refs(discard_pending, NULL, 0); + Assert(pgStatEntryRefHash->members == 0); + pgstat_entry_ref_hash_destroy(pgStatEntryRefHash); + pgStatEntryRefHash = NULL; +} + +static bool +match_db(PgStat_EntryRefHashEntry *ent, Datum match_data) +{ + Oid dboid = DatumGetObjectId(match_data); + + return ent->key.dboid == dboid; +} + +static void +pgstat_release_db_entry_refs(Oid dboid) +{ + pgstat_release_matching_entry_refs( /* discard pending = */ true, + match_db, + ObjectIdGetDatum(dboid)); +} + + +/* ------------------------------------------------------------ + * Dropping and resetting of stats entries + * ------------------------------------------------------------ + */ + +static void +pgstat_free_entry(PgStatShared_HashEntry *shent, dshash_seq_status *hstat) +{ + dsa_pointer pdsa; + + /* + * Fetch dsa pointer before deleting entry - that way we can free the + * memory after releasing the lock. + */ + pdsa = shent->body; + + if (!hstat) + dshash_delete_entry(pgStatLocal.shared_hash, shent); + else + dshash_delete_current(hstat); + + /* + * XXX: Might be worth adding some frobbing of the allocation before + * freeing, to make it easier to detect use-after-free style bugs. + */ + dsa_free(pgStatLocal.dsa, pdsa); +} + +/* + * Helper for both pgstat_drop_database_and_contents() and + * pgstat_drop_entry(). If hstat is non-null delete the shared entry using + * dshash_delete_current(), otherwise use dshash_delete_entry(). In either + * case the entry needs to be already locked. + */ +static bool +pgstat_drop_entry_internal(PgStatShared_HashEntry *shent, dshash_seq_status *hstat) +{ + Assert(shent->body != InvalidDsaPointer); + + /* should already have released local reference */ + if (pgStatEntryRefHash) + Assert(!pgstat_entry_ref_hash_lookup(pgStatEntryRefHash, shent->key)); + + /* + * Signal that the entry is dropped - this will eventually cause other + * backends to release their references. + */ + if (shent->dropped) + elog(ERROR, "can only drop stats once"); + shent->dropped = true; + + /* release refcount marking entry as not dropped */ + if (pg_atomic_sub_fetch_u32(&shent->refcount, 1) == 0) + { + pgstat_free_entry(shent, hstat); + return true; + } + else + { + if (!hstat) + dshash_release_lock(pgStatLocal.shared_hash, shent); + return false; + } +} + +/* + * Drop stats for the database and all the objects inside that database. + */ +static void +pgstat_drop_database_and_contents(Oid dboid) +{ + dshash_seq_status hstat; + PgStatShared_HashEntry *p; + uint64 not_freed_count = 0; + + Assert(OidIsValid(dboid)); + + Assert(pgStatLocal.shared_hash != NULL); + + /* + * This backend might very well be the only backend holding a reference to + * about-to-be-dropped entries. Ensure that we're not preventing it from + * being cleaned up till later. + * + * Doing this separately from the dshash iteration below avoids having to + * do so while holding a partition lock on the shared hashtable. + */ + pgstat_release_db_entry_refs(dboid); + + /* some of the dshash entries are to be removed, take exclusive lock. */ + dshash_seq_init(&hstat, pgStatLocal.shared_hash, true); + while ((p = dshash_seq_next(&hstat)) != NULL) + { + if (p->dropped) + continue; + + if (p->key.dboid != dboid) + continue; + + if (!pgstat_drop_entry_internal(p, &hstat)) + { + /* + * Even statistics for a dropped database might currently be + * accessed (consider e.g. database stats for pg_stat_database). + */ + not_freed_count++; + } + } + dshash_seq_term(&hstat); + + /* + * If some of the stats data could not be freed, signal the reference + * holders to run garbage collection of their cached pgStatShmLookupCache. + */ + if (not_freed_count > 0) + pgstat_request_entry_refs_gc(); +} + +bool +pgstat_drop_entry(PgStat_Kind kind, Oid dboid, Oid objoid) +{ + PgStat_HashKey key = {.kind = kind,.dboid = dboid,.objoid = objoid}; + PgStatShared_HashEntry *shent; + bool freed = true; + + /* delete local reference */ + if (pgStatEntryRefHash) + { + PgStat_EntryRefHashEntry *lohashent = + pgstat_entry_ref_hash_lookup(pgStatEntryRefHash, key); + + if (lohashent) + pgstat_release_entry_ref(lohashent->key, lohashent->entry_ref, true); + } + + /* mark entry in shared hashtable as deleted, drop if possible */ + shent = dshash_find(pgStatLocal.shared_hash, &key, true); + if (shent) + { + freed = pgstat_drop_entry_internal(shent, NULL); + + /* + * Database stats contain other stats. Drop those as well when + * dropping the database. XXX: Perhaps this should be done in a + * slightly more principled way? But not obvious what that'd look + * like, and so far this is the only case... + */ + if (key.kind == PGSTAT_KIND_DATABASE) + pgstat_drop_database_and_contents(key.dboid); + } + + return freed; +} + +/* + * The length of the data portion of a shared memory stats entry (i.e. without + * transient data such as refcoutns, lwlocks, ...). + */ +static inline size_t +shared_stat_entry_len(PgStat_Kind kind) +{ + return pgstat_kind_info_for(kind)->shared_data_len; +} + +/* + * Returns a pointer to the data portion of a shared memory stats entry. + */ +static inline void * +shared_stat_entry_data(PgStat_Kind kind, PgStatShared_Common *entry) +{ + size_t off = pgstat_kind_info_for(kind)->shared_data_off; + + Assert(off != 0 && off < PG_UINT32_MAX); + + return ((char *) (entry)) + off; +} + +static void +shared_stat_reset_contents(PgStat_Kind kind, PgStatShared_Common *header) +{ + const PgStat_KindInfo *kind_info = pgstat_kind_info_for(kind); + TimestampTz ts = GetCurrentTimestamp(); + + memset(shared_stat_entry_data(kind, header), 0, + shared_stat_entry_len(kind)); + + if (kind_info->reset_timestamp_cb) + kind_info->reset_timestamp_cb(header, ts); +} + +/* + * Reset one non-global shared stat entry. + */ +void +pgstat_reset_entry(PgStat_Kind kind, Oid dboid, Oid objoid) +{ + PgStat_EntryRef *entry_ref; + + Assert(!pgstat_kind_info_for(kind)->fixed_amount); + + entry_ref = pgstat_get_entry_ref(kind, dboid, objoid, false, NULL); + if (!entry_ref || entry_ref->shared_entry->dropped) + return; + + pgstat_lock_entry(entry_ref, false); + shared_stat_reset_contents(kind, entry_ref->shared_stats); + pgstat_unlock_entry(entry_ref); +} + +/* + * Scan through the shared hashtable of stats, resetting statistics if + * approved by the provided do_reset() function. + */ +void +pgstat_reset_matching_entries(bool (*do_reset) (PgStatShared_HashEntry *, Datum), Datum match_data) +{ + dshash_seq_status hstat; + PgStatShared_HashEntry *p; + + /* dshash entry is not modified, take shared lock */ + dshash_seq_init(&hstat, pgStatLocal.shared_hash, false); + while ((p = dshash_seq_next(&hstat)) != NULL) + { + PgStatShared_Common *header; + + if (p->dropped) + continue; + + if (!do_reset(p, match_data)) + continue; + + header = dsa_get_address(pgStatLocal.dsa, p->body); + + LWLockAcquire(&header->lock, LW_EXCLUSIVE); + + shared_stat_reset_contents(p->key.kind, header); + + LWLockRelease(&header->lock); + } + dshash_seq_term(&hstat); +} + +static bool +match_kind(PgStatShared_HashEntry *p, Datum match_data) +{ + return p->key.kind == DatumGetInt32(match_data); +} + +void +pgstat_reset_entries_of_kind(PgStat_Kind kind) +{ + pgstat_reset_matching_entries(match_kind, Int32GetDatum(kind)); +} + +static void +pgstat_setup_memcxt(void) +{ + if (unlikely(!pgStatSharedRefContext)) + pgStatSharedRefContext = + AllocSetContextCreate(CacheMemoryContext, + "PgStat Shared Ref", + ALLOCSET_SMALL_SIZES); + if (unlikely(!pgStatEntryRefHashContext)) + pgStatEntryRefHashContext = + AllocSetContextCreate(CacheMemoryContext, + "PgStat Shared Ref Hash", + ALLOCSET_SMALL_SIZES); +} diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 9cb9aeb890d..63810ec2db7 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -18,18 +18,21 @@ #include "postgres.h" #include "utils/pgstat_internal.h" +#include "utils/timestamp.h" -static inline PgStat_MsgSLRU *slru_entry(int slru_idx); +static inline PgStat_SLRUStats *slru_entry(int slru_idx); +static void pgstat_reset_slru_counter_internal(int index, TimestampTz ts); /* - * SLRU statistics counts waiting to be sent to the collector. These are - * stored directly in stats message format so they can be sent without needing - * to copy things around. We assume this variable inits to zeroes. Entries - * are one-to-one with slru_names[]. + * SLRU statistics counts waiting to be flushed out. We assume this variable + * inits to zeroes. Entries are one-to-one with slru_names[]. Changes of + * SLRU counters are reported within critical sections so we use static memory + * in order to avoid memory allocation. */ -static PgStat_MsgSLRU SLRUStats[SLRU_NUM_ELEMENTS]; +static PgStat_SLRUStats pending_SLRUStats[SLRU_NUM_ELEMENTS]; +bool have_slrustats = false; /* @@ -41,17 +44,11 @@ static PgStat_MsgSLRU SLRUStats[SLRU_NUM_ELEMENTS]; void pgstat_reset_slru_counter(const char *name) { - PgStat_MsgResetslrucounter msg; + TimestampTz ts = GetCurrentTimestamp(); AssertArg(name != NULL); - if (pgStatSock == PGINVALID_SOCKET) - return; - - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSLRUCOUNTER); - msg.m_index = pgstat_slru_index(name); - - pgstat_send(&msg, sizeof(msg)); + pgstat_reset_slru_counter_internal(pgstat_slru_index(name), ts); } /* @@ -61,43 +58,55 @@ pgstat_reset_slru_counter(const char *name) void pgstat_count_slru_page_zeroed(int slru_idx) { - slru_entry(slru_idx)->m_blocks_zeroed += 1; + slru_entry(slru_idx)->blocks_zeroed += 1; } void pgstat_count_slru_page_hit(int slru_idx) { - slru_entry(slru_idx)->m_blocks_hit += 1; + slru_entry(slru_idx)->blocks_hit += 1; } void pgstat_count_slru_page_exists(int slru_idx) { - slru_entry(slru_idx)->m_blocks_exists += 1; + slru_entry(slru_idx)->blocks_exists += 1; } void pgstat_count_slru_page_read(int slru_idx) { - slru_entry(slru_idx)->m_blocks_read += 1; + slru_entry(slru_idx)->blocks_read += 1; } void pgstat_count_slru_page_written(int slru_idx) { - slru_entry(slru_idx)->m_blocks_written += 1; + slru_entry(slru_idx)->blocks_written += 1; } void pgstat_count_slru_flush(int slru_idx) { - slru_entry(slru_idx)->m_flush += 1; + slru_entry(slru_idx)->flush += 1; } void pgstat_count_slru_truncate(int slru_idx) { - slru_entry(slru_idx)->m_truncate += 1; + slru_entry(slru_idx)->truncate += 1; +} + +/* + * Support function for the SQL-callable pgstat* functions. Returns + * a pointer to the slru statistics struct. + */ +PgStat_SLRUStats * +pgstat_fetch_slru(void) +{ + pgstat_snapshot_global(PGSTAT_KIND_SLRU); + + return pgStatLocal.snapshot.slru; } /* @@ -135,45 +144,83 @@ pgstat_slru_index(const char *name) } /* - * Send SLRU statistics to the collector + * Flush out locally pending SLRU stats entries + * + * If nowait is true, this function returns false on lock failure. Otherwise + * this function always returns true. Writer processes are mutually excluded + * using LWLock, but readers are expected to use change-count protocol to avoid + * interference with writers. + * + * If nowait is true, this function returns true if the lock could not be + * acquired. Otherwise return false. */ -void -pgstat_send_slru(void) +bool +pgstat_slru_flush(bool nowait) { - /* We assume this initializes to zeroes */ - static const PgStat_MsgSLRU all_zeroes; + PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; + int i; - for (int i = 0; i < SLRU_NUM_ELEMENTS; i++) + if (!have_slrustats) + return false; + + /* lock the shared entry to protect the content, skip if failed */ + if (!nowait) + LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE); + else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE)) + return true; /* failed to acquire lock, skip */ + + + for (i = 0; i < SLRU_NUM_ELEMENTS; i++) { - /* - * This function can be called even if nothing at all has happened. In - * this case, avoid sending a completely empty message to the stats - * collector. - */ - if (memcmp(&SLRUStats[i], &all_zeroes, sizeof(PgStat_MsgSLRU)) == 0) - continue; + PgStat_SLRUStats *sharedent = &stats_shmem->stats[i]; + PgStat_SLRUStats *pendingent = &pending_SLRUStats[i]; - /* set the SLRU type before each send */ - SLRUStats[i].m_index = i; - - /* - * Prepare and send the message - */ - pgstat_setheader(&SLRUStats[i].m_hdr, PGSTAT_MTYPE_SLRU); - pgstat_send(&SLRUStats[i], sizeof(PgStat_MsgSLRU)); - - /* - * Clear out the statistics buffer, so it can be re-used. - */ - MemSet(&SLRUStats[i], 0, sizeof(PgStat_MsgSLRU)); +#define SLRU_ACC(fld) sharedent->fld += pendingent->fld + SLRU_ACC(blocks_zeroed); + SLRU_ACC(blocks_hit); + SLRU_ACC(blocks_read); + SLRU_ACC(blocks_written); + SLRU_ACC(blocks_exists); + SLRU_ACC(flush); + SLRU_ACC(truncate); +#undef SLRU_ACC } + + /* done, clear the pending entry */ + MemSet(pending_SLRUStats, 0, sizeof(pending_SLRUStats)); + + LWLockRelease(&stats_shmem->lock); + + have_slrustats = false; + + return false; +} + +void +pgstat_slru_reset_all_cb(TimestampTz now) +{ + for (int i = 0; i < SLRU_NUM_ELEMENTS; i++) + pgstat_reset_slru_counter_internal(i, now); +} + +void +pgstat_slru_snapshot_cb(void) +{ + PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; + + LWLockAcquire(&stats_shmem->lock, LW_SHARED); + + memcpy(pgStatLocal.snapshot.slru, &stats_shmem->stats, + sizeof(stats_shmem->stats)); + + LWLockRelease(&stats_shmem->lock); } /* * Returns pointer to entry with counters for given SLRU (based on the name * stored in SlruCtl as lwlock tranche name). */ -static inline PgStat_MsgSLRU * +static inline PgStat_SLRUStats * slru_entry(int slru_idx) { pgstat_assert_is_up(); @@ -186,5 +233,20 @@ slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); - return &SLRUStats[slru_idx]; + have_slrustats = true; + + return &pending_SLRUStats[slru_idx]; +} + +static void +pgstat_reset_slru_counter_internal(int index, TimestampTz ts) +{ + PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; + + LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE); + + memset(&stats_shmem->stats[index], 0, sizeof(PgStat_SLRUStats)); + stats_shmem->stats[index].stat_reset_timestamp = ts; + + LWLockRelease(&stats_shmem->lock); } diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index d1d8c0ecfd2..ec47ab26455 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -18,6 +18,7 @@ #include "postgres.h" #include "utils/pgstat_internal.h" +#include "utils/timestamp.h" /* @@ -26,14 +27,7 @@ void pgstat_reset_subscription_counters(void) { - PgStat_MsgResetsubcounter msg; - - if (pgStatSock == PGINVALID_SOCKET) - return; - - msg.m_subid = InvalidOid; - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSUBCOUNTER); - pgstat_send(&msg, sizeof(msg)); + pgstat_reset_entries_of_kind(PGSTAT_KIND_SUBSCRIPTION); } /* @@ -45,14 +39,7 @@ pgstat_reset_subscription_counters(void) void pgstat_reset_subscription_counter(Oid subid) { - PgStat_MsgResetsubcounter msg; - - if (pgStatSock == PGINVALID_SOCKET) - return; - - msg.m_subid = subid; - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSUBCOUNTER); - pgstat_send(&msg, sizeof(msg)); + pgstat_reset_entry(PGSTAT_KIND_SUBSCRIPTION, InvalidOid, subid); } /* @@ -61,12 +48,17 @@ pgstat_reset_subscription_counter(Oid subid) void pgstat_report_subscription_error(Oid subid, bool is_apply_error) { - PgStat_MsgSubscriptionError msg; + PgStat_EntryRef *entry_ref; + PgStat_BackendSubEntry *pending; - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_SUBSCRIPTIONERROR); - msg.m_subid = subid; - msg.m_is_apply_error = is_apply_error; - pgstat_send(&msg, sizeof(PgStat_MsgSubscriptionError)); + entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_SUBSCRIPTION, + InvalidOid, subid, NULL); + pending = entry_ref->pending; + + if (is_apply_error) + pending->apply_error_count++; + else + pending->sync_error_count++; } /* @@ -89,12 +81,52 @@ pgstat_create_subscription(Oid subid) void pgstat_drop_subscription(Oid subid) { - PgStat_MsgSubscriptionDrop msg; - - pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_SUBSCRIPTIONDROP); - msg.m_subid = subid; - pgstat_send(&msg, sizeof(PgStat_MsgSubscriptionDrop)); - pgstat_drop_transactional(PGSTAT_KIND_SUBSCRIPTION, InvalidOid, subid); } + +/* + * Support function for the SQL-callable pgstat* functions. Returns + * the collected statistics for one subscription or NULL. + */ +PgStat_StatSubEntry * +pgstat_fetch_stat_subscription(Oid subid) +{ + return (PgStat_StatSubEntry *) + pgstat_fetch_entry(PGSTAT_KIND_SUBSCRIPTION, InvalidOid, subid); +} + +/* + * Flush out pending stats for the entry + * + * If nowait is true, this function returns false if lock could not + * immediately acquired, otherwise true is returned. + */ +bool +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +{ + PgStat_BackendSubEntry *localent; + PgStatShared_Subscription *shsubent; + + localent = (PgStat_BackendSubEntry *) entry_ref->pending; + shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; + + /* localent always has non-zero content */ + + if (!pgstat_lock_entry(entry_ref, nowait)) + return false; + +#define SUB_ACC(fld) shsubent->stats.fld += localent->fld + SUB_ACC(apply_error_count); + SUB_ACC(sync_error_count); +#undef SUB_ACC + + pgstat_unlock_entry(entry_ref); + return true; +} + +void +pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts) +{ + ((PgStatShared_Subscription *) header)->stats.stat_reset_timestamp = ts; +} diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 8680809deef..ceb6709aada 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -21,13 +21,7 @@ #include "executor/instrument.h" -/* - * WAL global statistics counters. Stored directly in a stats message - * structure so they can be sent without needing to copy things around. We - * assume these init to zeroes. - */ -PgStat_MsgWal WalStats; - +PgStat_WalStats PendingWalStats = {0}; /* * WAL usage counters saved from pgWALUsage at the previous call to @@ -39,101 +33,99 @@ static WalUsage prevWalUsage; /* - * Send WAL statistics to the collector. + * Calculate how much WAL usage counters have increased and update + * shared statistics. * - * If 'force' is not set, WAL stats message is only sent if enough time has - * passed since last one was sent to reach PGSTAT_STAT_INTERVAL. + * Must be called by processes that generate WAL. */ void pgstat_report_wal(bool force) { - static TimestampTz sendTime = 0; + pgstat_flush_wal(force); +} + +/* + * Support function for the SQL-callable pgstat* functions. Returns + * a pointer to the WAL statistics struct. + */ +PgStat_WalStats * +pgstat_fetch_stat_wal(void) +{ + pgstat_snapshot_global(PGSTAT_KIND_WAL); + + return &pgStatLocal.snapshot.wal; +} + +/* + * Calculate how much WAL usage counters have increased by substracting the + * previous counters from the current ones. + * + * If nowait is true, this function returns true if the lock could not be + * acquired. Otherwise return false. + */ +bool +pgstat_flush_wal(bool nowait) +{ + WalUsage diff = {0}; + + Assert(IsUnderPostmaster || !IsPostmasterEnvironment); + Assert(pgStatLocal.shmem != NULL && + !pgStatLocal.shmem->is_shutdown); /* - * This function can be called even if nothing at all has happened. In - * this case, avoid sending a completely empty message to the stats - * collector. - * - * Check wal_records counter to determine whether any WAL activity has - * happened since last time. Note that other WalUsage counters don't need - * to be checked because they are incremented always together with - * wal_records counter. - * - * m_wal_buffers_full also doesn't need to be checked because it's - * incremented only when at least one WAL record is generated (i.e., - * wal_records counter is incremented). But for safely, we assert that - * m_wal_buffers_full is always zero when no WAL record is generated - * - * This function can be called by a process like walwriter that normally - * generates no WAL records. To determine whether any WAL activity has - * happened at that process since the last time, the numbers of WAL writes - * and syncs are also checked. + * This function can be called even if nothing at all has happened. Avoid + * taking lock for nothing in that case. */ - if (pgWalUsage.wal_records == prevWalUsage.wal_records && - WalStats.m_wal_write == 0 && WalStats.m_wal_sync == 0) - { - Assert(WalStats.m_wal_buffers_full == 0); - return; - } - - if (!force) - { - TimestampTz now = GetCurrentTimestamp(); - - /* - * Don't send a message unless it's been at least PGSTAT_STAT_INTERVAL - * msec since we last sent one to avoid overloading the stats - * collector. - */ - if (!TimestampDifferenceExceeds(sendTime, now, PGSTAT_STAT_INTERVAL)) - return; - sendTime = now; - } + if (!pgstat_wal_pending()) + return false; /* - * Set the counters related to generated WAL data if the counters were - * updated. + * We don't update the WAL usage portion of the local WalStats elsewhere. + * Calculate how much WAL usage counters were increased by subtracting the + * previous counters from the current ones. */ - if (pgWalUsage.wal_records != prevWalUsage.wal_records) - { - WalUsage walusage; + WalUsageAccumDiff(&diff, &pgWalUsage, &prevWalUsage); + PendingWalStats.wal_records = diff.wal_records; + PendingWalStats.wal_fpi = diff.wal_fpi; + PendingWalStats.wal_bytes = diff.wal_bytes; - /* - * Calculate how much WAL usage counters were increased by subtracting - * the previous counters from the current ones. Fill the results in - * WAL stats message. - */ - MemSet(&walusage, 0, sizeof(WalUsage)); - WalUsageAccumDiff(&walusage, &pgWalUsage, &prevWalUsage); + /* lock the shared entry to protect the content, skip if failed */ + if (!nowait) + LWLockAcquire(&pgStatLocal.shmem->wal.lock, LW_EXCLUSIVE); + else if (!LWLockConditionalAcquire(&pgStatLocal.shmem->wal.lock, LW_EXCLUSIVE)) + return true; /* failed to acquire lock, skip */ - WalStats.m_wal_records = walusage.wal_records; - WalStats.m_wal_fpi = walusage.wal_fpi; - WalStats.m_wal_bytes = walusage.wal_bytes; +#define WALSTAT_ACC(fld) pgStatLocal.shmem->wal.stats.fld += PendingWalStats.fld + WALSTAT_ACC(wal_records); + WALSTAT_ACC(wal_fpi); + WALSTAT_ACC(wal_bytes); + WALSTAT_ACC(wal_buffers_full); + WALSTAT_ACC(wal_write); + WALSTAT_ACC(wal_sync); + WALSTAT_ACC(wal_write_time); + WALSTAT_ACC(wal_sync_time); +#undef WALSTAT_ACC - /* - * Save the current counters for the subsequent calculation of WAL - * usage. - */ - prevWalUsage = pgWalUsage; - } + LWLockRelease(&pgStatLocal.shmem->wal.lock); /* - * Prepare and send the message + * Save the current counters for the subsequent calculation of WAL usage. */ - pgstat_setheader(&WalStats.m_hdr, PGSTAT_MTYPE_WAL); - pgstat_send(&WalStats, sizeof(WalStats)); + prevWalUsage = pgWalUsage; /* * Clear out the statistics buffer, so it can be re-used. */ - MemSet(&WalStats, 0, sizeof(WalStats)); + MemSet(&PendingWalStats, 0, sizeof(PendingWalStats)); + + return false; } void pgstat_wal_initialize(void) { /* - * Initialize prevWalUsage with pgWalUsage so that pgstat_report_wal() can + * Initialize prevWalUsage with pgWalUsage so that pgstat_flush_wal() can * calculate how much pgWalUsage counters are increased by subtracting * prevWalUsage from pgWalUsage. */ @@ -151,6 +143,28 @@ bool pgstat_wal_pending(void) { return pgWalUsage.wal_records != prevWalUsage.wal_records || - WalStats.m_wal_write != 0 || - WalStats.m_wal_sync != 0; + PendingWalStats.wal_write != 0 || + PendingWalStats.wal_sync != 0; +} + +void +pgstat_wal_reset_all_cb(TimestampTz now) +{ + PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; + + LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE); + memset(&stats_shmem->stats, 0, sizeof(stats_shmem->stats)); + stats_shmem->stats.stat_reset_timestamp = now; + LWLockRelease(&stats_shmem->lock); +} + +void +pgstat_wal_snapshot_cb(void) +{ + PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; + + LWLockAcquire(&stats_shmem->lock, LW_SHARED); + memcpy(&pgStatLocal.snapshot.wal, &stats_shmem->stats, + sizeof(pgStatLocal.snapshot.wal)); + LWLockRelease(&stats_shmem->lock); } diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c index 5d3c184efb2..a1697ff8b37 100644 --- a/src/backend/utils/activity/pgstat_xact.c +++ b/src/backend/utils/activity/pgstat_xact.c @@ -68,6 +68,7 @@ static void AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit) { dlist_mutable_iter iter; + int not_freed_count = 0; if (xact_state->pending_drops_count == 0) { @@ -79,6 +80,7 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit) { PgStat_PendingDroppedStatsItem *pending = dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur); + xl_xact_stats_item *it = &pending->item; if (isCommit && !pending->is_create) { @@ -86,7 +88,8 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit) * Transaction that dropped an object committed. Drop the stats * too. */ - /* will do work in subsequent commit */ + if (!pgstat_drop_entry(it->kind, it->dboid, it->objoid)) + not_freed_count++; } else if (!isCommit && pending->is_create) { @@ -94,13 +97,17 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit) * Transaction that created an object aborted. Drop the stats * associated with the object. */ - /* will do work in subsequent commit */ + if (!pgstat_drop_entry(it->kind, it->dboid, it->objoid)) + not_freed_count++; } dlist_delete(&pending->node); xact_state->pending_drops_count--; pfree(pending); } + + if (not_freed_count > 0) + pgstat_request_entry_refs_gc(); } /* @@ -135,6 +142,7 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, { PgStat_SubXactStatus *parent_xact_state; dlist_mutable_iter iter; + int not_freed_count = 0; if (xact_state->pending_drops_count == 0) return; @@ -145,6 +153,7 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, { PgStat_PendingDroppedStatsItem *pending = dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur); + xl_xact_stats_item *it = &pending->item; dlist_delete(&pending->node); xact_state->pending_drops_count--; @@ -155,7 +164,8 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, * Subtransaction creating a new stats object aborted. Drop the * stats object. */ - /* will do work in subsequent commit */ + if (!pgstat_drop_entry(it->kind, it->dboid, it->objoid)) + not_freed_count++; pfree(pending); } else if (isCommit) @@ -175,6 +185,8 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, } Assert(xact_state->pending_drops_count == 0); + if (not_freed_count > 0) + pgstat_request_entry_refs_gc(); } /* @@ -304,13 +316,21 @@ pgstat_get_transactional_drops(bool isCommit, xl_xact_stats_item **items) void pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_item *items, bool is_redo) { + int not_freed_count = 0; + if (ndrops == 0) return; for (int i = 0; i < ndrops; i++) { - /* will do work in subsequent commit */ + xl_xact_stats_item *it = &items[i]; + + if (!pgstat_drop_entry(it->kind, it->dboid, it->objoid)) + not_freed_count++; } + + if (not_freed_count > 0) + pgstat_request_entry_refs_gc(); } static void @@ -342,6 +362,17 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, Oid objoid, bool void pgstat_create_transactional(PgStat_Kind kind, Oid dboid, Oid objoid) { + if (pgstat_get_entry_ref(kind, dboid, objoid, false, NULL)) + { + Oid msg_oid = (kind == PGSTAT_KIND_DATABASE) ? dboid : objoid; + + ereport(WARNING, + errmsg("Resetting existing stats for %s with OID %d.", + (pgstat_kind_info_for(kind))->name, msg_oid)); + + pgstat_reset_entry(kind, dboid, objoid); + } + create_drop_transactional_internal(kind, dboid, objoid, /* create */ true); } diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 1c8aba49259..87c15b9c6f3 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -230,9 +230,6 @@ pgstat_get_wait_activity(WaitEventActivity w) case WAIT_EVENT_LOGICAL_LAUNCHER_MAIN: event_name = "LogicalLauncherMain"; break; - case WAIT_EVENT_PGSTAT_MAIN: - event_name = "PgStatMain"; - break; case WAIT_EVENT_RECOVERY_WAL_STREAM: event_name = "RecoveryWalStream"; break; diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 02676bf01dc..848f410c96b 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -2047,7 +2047,15 @@ pg_stat_get_xact_function_self_time(PG_FUNCTION_ARGS) Datum pg_stat_get_snapshot_timestamp(PG_FUNCTION_ARGS) { - PG_RETURN_TIMESTAMPTZ(pgstat_fetch_global()->stats_timestamp); + bool have_snapshot; + TimestampTz ts; + + ts = pgstat_get_stat_snapshot_timestamp(&have_snapshot); + + if (!have_snapshot) + PG_RETURN_NULL(); + + PG_RETURN_TIMESTAMPTZ(ts); } /* Discard the active statistics snapshot */ diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index a15ce9edb13..294395058e0 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -73,6 +73,7 @@ #include "nodes/makefuncs.h" #include "nodes/nodeFuncs.h" #include "optimizer/optimizer.h" +#include "pgstat.h" #include "rewrite/rewriteDefine.h" #include "rewrite/rowsecurity.h" #include "storage/lmgr.h" @@ -2409,6 +2410,9 @@ RelationDestroyRelation(Relation relation, bool remember_tupdesc) */ RelationCloseSmgr(relation); + /* break mutual link with stats entry */ + pgstat_relation_unlink(relation); + /* * Free all the subsidiary data structures of the relcache entry, then the * entry itself. @@ -2718,6 +2722,7 @@ RelationClearRelation(Relation relation, bool rebuild) SWAPFIELD(Oid, rd_toastoid); /* pgstat_info must be preserved */ SWAPFIELD(struct PgStat_TableStatus *, pgstat_info); + SWAPFIELD(bool, pgstat_enabled); /* preserve old partition key if we have one */ if (keep_partkey) { diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 3419c099b28..1a5d29ac9ba 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -36,6 +36,7 @@ volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false; volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; +volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c index bdc77af7194..0d3cfe8240b 100644 --- a/src/backend/utils/init/miscinit.c +++ b/src/backend/utils/init/miscinit.c @@ -288,9 +288,6 @@ GetBackendTypeDesc(BackendType backendType) case B_ARCHIVER: backendDesc = "archiver"; break; - case B_STATS_COLLECTOR: - backendDesc = "stats collector"; - break; case B_LOGGER: backendDesc = "logger"; break; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 6452b42dbff..824bc97b031 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -80,6 +80,7 @@ static void StatementTimeoutHandler(void); static void LockTimeoutHandler(void); static void IdleInTransactionSessionTimeoutHandler(void); static void IdleSessionTimeoutHandler(void); +static void IdleStatsUpdateTimeoutHandler(void); static void ClientCheckTimeoutHandler(void); static bool ThereIsAtLeastOneRole(void); static void process_startup_options(Port *port, bool am_superuser); @@ -725,6 +726,8 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username, IdleInTransactionSessionTimeoutHandler); RegisterTimeout(IDLE_SESSION_TIMEOUT, IdleSessionTimeoutHandler); RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); + RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, + IdleStatsUpdateTimeoutHandler); } /* @@ -752,6 +755,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username, * Use before_shmem_exit() so that ShutdownXLOG() can rely on DSM * segments etc to work (which in turn is required for pgstats). */ + before_shmem_exit(pgstat_before_server_shutdown, 0); before_shmem_exit(ShutdownXLOG, 0); } @@ -1352,6 +1356,14 @@ IdleSessionTimeoutHandler(void) SetLatch(MyLatch); } +static void +IdleStatsUpdateTimeoutHandler(void) +{ + IdleStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} + static void ClientCheckTimeoutHandler(void) { diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 9e8ab1420d9..bbb156e0dfe 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -374,6 +374,16 @@ static const struct config_enum_entry track_function_options[] = { StaticAssertDecl(lengthof(track_function_options) == (TRACK_FUNC_ALL + 2), "array length mismatch"); +static const struct config_enum_entry stats_fetch_consistency[] = { + {"none", PGSTAT_FETCH_CONSISTENCY_NONE, false}, + {"cache", PGSTAT_FETCH_CONSISTENCY_CACHE, false}, + {"snapshot", PGSTAT_FETCH_CONSISTENCY_SNAPSHOT, false}, + {NULL, 0, false} +}; + +StaticAssertDecl(lengthof(stats_fetch_consistency) == (PGSTAT_FETCH_CONSISTENCY_SNAPSHOT + 2), + "array length mismatch"); + static const struct config_enum_entry xmlbinary_options[] = { {"base64", XMLBINARY_BASE64, false}, {"hex", XMLBINARY_HEX, false}, @@ -4917,6 +4927,17 @@ static struct config_enum ConfigureNamesEnum[] = NULL, NULL, NULL }, + + { + {"stats_fetch_consistency", PGC_USERSET, STATS_COLLECTOR, + gettext_noop("Sets the consistency of accesses to statistics data"), + NULL + }, + &pgstat_fetch_consistency, + PGSTAT_FETCH_CONSISTENCY_CACHE, stats_fetch_consistency, + NULL, NULL, NULL + }, + { {"wal_compression", PGC_SUSET, WAL_SETTINGS, gettext_noop("Compresses full-page writes written in WAL file with specified method."), diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 93d221a37b1..5f9a37bed3b 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -614,6 +614,7 @@ #track_wal_io_timing = off #track_functions = none # none, pl, all #stats_temp_directory = 'pg_stat_tmp' +#stats_fetch_consistency = none # - Monitoring - diff --git a/contrib/test_decoding/expected/stats.out b/contrib/test_decoding/expected/stats.out index 206c0a126e5..a10f8214ce2 100644 --- a/contrib/test_decoding/expected/stats.out +++ b/contrib/test_decoding/expected/stats.out @@ -120,6 +120,7 @@ SELECT slot_name, spill_txns > 0 AS spill_txns, spill_count > 0 AS spill_count F -- Ensure stats can be repeatedly accessed using the same stats snapshot. See -- https://postgr.es/m/20210317230447.c7uc4g3vbs4wi32i%40alap3.anarazel.de BEGIN; +SET LOCAL stats_fetch_consistency = snapshot; SELECT slot_name FROM pg_stat_replication_slots; slot_name ----------------------- diff --git a/contrib/test_decoding/sql/stats.sql b/contrib/test_decoding/sql/stats.sql index 67462ca27f7..08616a2f965 100644 --- a/contrib/test_decoding/sql/stats.sql +++ b/contrib/test_decoding/sql/stats.sql @@ -78,6 +78,7 @@ SELECT slot_name, spill_txns > 0 AS spill_txns, spill_count > 0 AS spill_count F -- Ensure stats can be repeatedly accessed using the same stats snapshot. See -- https://postgr.es/m/20210317230447.c7uc4g3vbs4wi32i%40alap3.anarazel.de BEGIN; +SET LOCAL stats_fetch_consistency = snapshot; SELECT slot_name FROM pg_stat_replication_slots; SELECT slot_name FROM pg_stat_replication_slots; COMMIT; diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index 3b9172f65bd..7c9469fb447 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -1110,10 +1110,6 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser LogicalLauncherMain Waiting in main loop of logical replication launcher process. - - PgStatMain - Waiting in main loop of statistics collector process. - RecoveryWalStream Waiting in main loop of startup process for WAL to arrive, during @@ -2115,6 +2111,18 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser Waiting to access the list of predicate locks held by serializable transactions. + + PgStatsDSA + Waiting for stats dynamic shared memory allocator access + + + PgStatsHash + Waiting for stats shared memory hash table access + + + PgStatsData + Waiting fo shared memory stats data access + SerializableXactHash Waiting to read or update information about serializable @@ -5142,7 +5150,8 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i timestamp with time zone - Returns the timestamp of the current statistics snapshot. + Returns the timestamp of the current statistics snapshot, or NULL if + no statistics snapshot has been taken. diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out index 57550d49e77..2f824258cd6 100644 --- a/src/test/regress/expected/stats.out +++ b/src/test/regress/expected/stats.out @@ -17,6 +17,8 @@ SET enable_indexscan TO on; -- for the moment, we don't want index-only scans here SET enable_indexonlyscan TO off; -- save counters +BEGIN; +SET LOCAL stats_fetch_consistency = snapshot; CREATE TABLE prevstats AS SELECT t.seq_scan, t.seq_tup_read, t.idx_scan, t.idx_tup_fetch, (b.heap_blks_read + b.heap_blks_hit) AS heap_blks, @@ -25,6 +27,7 @@ SELECT t.seq_scan, t.seq_tup_read, t.idx_scan, t.idx_tup_fetch, FROM pg_catalog.pg_stat_user_tables AS t, pg_catalog.pg_statio_user_tables AS b WHERE t.relname='tenk2' AND b.relname='tenk2'; +COMMIT; -- function to wait for counters to advance create function wait_for_stats() returns void as $$ declare @@ -34,6 +37,8 @@ declare updated3 bool; updated4 bool; begin + SET LOCAL stats_fetch_consistency = snapshot; + -- We don't want to wait forever. No timeout suffices if the OS drops our -- stats traffic because an earlier test file left a full UDP buffer. -- Hence, don't use PG_TEST_TIMEOUT_DEFAULT, which may be large for @@ -163,6 +168,8 @@ SELECT wait_for_stats(); (1 row) -- check effects +BEGIN; +SET LOCAL stats_fetch_consistency = snapshot; SELECT relname, n_tup_ins, n_tup_upd, n_tup_del, n_live_tup, n_dead_tup FROM pg_stat_user_tables WHERE relname like 'trunc_stats_test%' order by relname; @@ -202,6 +209,7 @@ FROM prevstats AS pr; t (1 row) +COMMIT; DROP TABLE trunc_stats_test, trunc_stats_test1, trunc_stats_test2, trunc_stats_test3, trunc_stats_test4; DROP TABLE prevstats; -- test BRIN index doesn't block HOT update - we include this test here, as it diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql index 7e746dca3b7..4f33ea6a7c8 100644 --- a/src/test/regress/sql/stats.sql +++ b/src/test/regress/sql/stats.sql @@ -15,6 +15,8 @@ SET enable_indexscan TO on; SET enable_indexonlyscan TO off; -- save counters +BEGIN; +SET LOCAL stats_fetch_consistency = snapshot; CREATE TABLE prevstats AS SELECT t.seq_scan, t.seq_tup_read, t.idx_scan, t.idx_tup_fetch, (b.heap_blks_read + b.heap_blks_hit) AS heap_blks, @@ -23,6 +25,7 @@ SELECT t.seq_scan, t.seq_tup_read, t.idx_scan, t.idx_tup_fetch, FROM pg_catalog.pg_stat_user_tables AS t, pg_catalog.pg_statio_user_tables AS b WHERE t.relname='tenk2' AND b.relname='tenk2'; +COMMIT; -- function to wait for counters to advance create function wait_for_stats() returns void as $$ @@ -33,6 +36,8 @@ declare updated3 bool; updated4 bool; begin + SET LOCAL stats_fetch_consistency = snapshot; + -- We don't want to wait forever. No timeout suffices if the OS drops our -- stats traffic because an earlier test file left a full UDP buffer. -- Hence, don't use PG_TEST_TIMEOUT_DEFAULT, which may be large for @@ -158,6 +163,9 @@ RESET enable_bitmapscan; SELECT wait_for_stats(); -- check effects +BEGIN; +SET LOCAL stats_fetch_consistency = snapshot; + SELECT relname, n_tup_ins, n_tup_upd, n_tup_del, n_live_tup, n_dead_tup FROM pg_stat_user_tables WHERE relname like 'trunc_stats_test%' order by relname; @@ -177,6 +185,8 @@ SELECT st.heap_blks_read + st.heap_blks_hit >= pr.heap_blks + cl.relpages, SELECT pr.snap_ts < pg_stat_get_snapshot_timestamp() as snapshot_newer FROM prevstats AS pr; +COMMIT; + DROP TABLE trunc_stats_test, trunc_stats_test1, trunc_stats_test2, trunc_stats_test3, trunc_stats_test4; DROP TABLE prevstats; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index efb30600e0d..14d9e299652 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1922,51 +1922,39 @@ PgFdwPathExtraData PgFdwRelationInfo PgFdwScanState PgIfAddrCallback +PgStatShared_Archiver +PgStatShared_BgWriter +PgStatShared_Checkpointer +PgStatShared_Common +PgStatShared_Database +PgStatShared_Function +PgStatShared_HashEntry +PgStatShared_Relation +PgStatShared_ReplSlot +PgStatShared_SLRU +PgStatShared_Subscription +PgStatShared_Wal PgStat_ArchiverStats PgStat_BackendFunctionEntry +PgStat_BackendSubEntry PgStat_BgWriterStats PgStat_CheckpointerStats PgStat_Counter +PgStat_EntryRef +PgStat_EntryRefHashEntry +PgStat_FetchConsistency PgStat_FunctionCallUsage PgStat_FunctionCounts -PgStat_FunctionEntry -PgStat_GlobalStats +PgStat_HashKey PgStat_Kind -PgStat_Msg -PgStat_MsgAnalyze -PgStat_MsgAnlAncestors -PgStat_MsgArchiver -PgStat_MsgAutovacStart -PgStat_MsgBgWriter -PgStat_MsgCheckpointer -PgStat_MsgChecksumFailure -PgStat_MsgConnect -PgStat_MsgDeadlock -PgStat_MsgDisconnect -PgStat_MsgDropdb -PgStat_MsgDummy -PgStat_MsgFuncpurge -PgStat_MsgFuncstat -PgStat_MsgHdr -PgStat_MsgInquiry -PgStat_MsgRecoveryConflict -PgStat_MsgReplSlot -PgStat_MsgResetcounter -PgStat_MsgResetreplslotcounter -PgStat_MsgResetsharedcounter -PgStat_MsgResetsinglecounter -PgStat_MsgResetslrucounter -PgStat_MsgResetsubcounter -PgStat_MsgSLRU -PgStat_MsgSubscriptionDrop -PgStat_MsgSubscriptionError -PgStat_MsgTabpurge -PgStat_MsgTabstat -PgStat_MsgTempFile -PgStat_MsgVacuum -PgStat_MsgWal +PgStat_KindInfo +PgStat_LocalState PgStat_PendingDroppedStatsItem +PgStat_ReplSlotStats PgStat_SLRUStats +PgStat_ShmemControl +PgStat_Snapshot +PgStat_SnapshotEntry PgStat_StatDBEntry PgStat_StatFuncEntry PgStat_StatReplSlotEntry @@ -1974,7 +1962,6 @@ PgStat_StatSubEntry PgStat_StatTabEntry PgStat_SubXactStatus PgStat_TableCounts -PgStat_TableEntry PgStat_TableStatus PgStat_TableXactStatus PgStat_WalStats @@ -2522,7 +2509,6 @@ StartReplicationCmd StartupStatusEnum StatEntry StatExtEntry -StatMsgType StateFileChunk StatisticExtInfo Stats @@ -2636,8 +2622,6 @@ TXNEntryFile TYPCATEGORY T_Action T_WorkerStatus -TabStatHashEntry -TabStatusArray TableAmRoutine TableAttachInfo TableDataInfo @@ -3421,6 +3405,7 @@ pgssHashKey pgssSharedState pgssStoreKind pgssVersion +pgstat_entry_ref_hash_hash pgstat_page pgstattuple_type pgthreadlock_t -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0014-pgstat-add-pg_stat_force_next_flush.patch" From ff62948bbb7a2dac6d6297b5a749c98d211769a4 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sat, 2 Apr 2022 14:21:51 -0700 Subject: [PATCH v68 14/31] pgstat: add pg_stat_force_next_flush(). FIXME: Needs to bump catversion. Author: Andres Freund Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- src/include/catalog/pg_proc.dat | 5 +++++ src/include/pgstat.h | 1 + src/backend/postmaster/pgstat.c | 23 +++++++++++++++++++++++ src/backend/utils/adt/pgstatfuncs.c | 10 ++++++++++ 4 files changed, 39 insertions(+) diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 25304430f44..12526c599e0 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -5744,6 +5744,11 @@ proname => 'pg_stat_clear_snapshot', proisstrict => 'f', provolatile => 'v', proparallel => 'r', prorettype => 'void', proargtypes => '', prosrc => 'pg_stat_clear_snapshot' }, +{ oid => '2137', + descr => 'statistics: force stats to be flushed after the next commit', + proname => 'pg_stat_force_next_flush', proisstrict => 'f', provolatile => 'v', + proparallel => 'r', prorettype => 'void', proargtypes => '', + prosrc => 'pg_stat_force_next_flush' }, { oid => '2274', descr => 'statistics: reset collected statistics for current database', proname => 'pg_stat_reset', proisstrict => 'f', provolatile => 'v', diff --git a/src/include/pgstat.h b/src/include/pgstat.h index b4d1be8a79d..3481ba5e262 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -427,6 +427,7 @@ extern void pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_ /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); extern void pgstat_reset_single_counter(PgStat_Kind kind, Oid objectid); diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index 650edc0c8dd..0a0d64be829 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/postmaster/pgstat.c @@ -167,6 +167,12 @@ static MemoryContext pgStatPendingContext = NULL; static dlist_head pgStatPending = DLIST_STATIC_INIT(pgStatPending); +/* + * Force the next stats flush to happen regardless of + * PGSTAT_MIN_INTERVAL. Useful in test scripts. + */ +static bool pgStatForceNextFlush = false; + /* * For assertions that check pgstat is not used before initialization / after * shutdown. @@ -511,6 +517,13 @@ pgstat_report_stat(bool force) pgstat_assert_is_up(); + /* "absorb" the forced flush even if there's nothing to flush */ + if (pgStatForceNextFlush) + { + force = true; + pgStatForceNextFlush = false; + } + /* Don't expend a clock check if nothing to do */ if (dlist_is_empty(&pgStatPending) && !have_slrustats && @@ -609,6 +622,16 @@ pgstat_report_stat(bool force) return 0; } +/* + * Force locally pending stats to be flushed during the next + * pgstat_report_stat() call. This is useful for writing tests. + */ +void +pgstat_force_next_flush(void) +{ + pgStatForceNextFlush = true; +} + /* * Only for use by pgstat_reset_counters() */ diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 848f410c96b..caaf0c5a672 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -2068,6 +2068,16 @@ pg_stat_clear_snapshot(PG_FUNCTION_ARGS) } +/* Force statistics to be reported at the next occasion */ +Datum +pg_stat_force_next_flush(PG_FUNCTION_ARGS) +{ + pgstat_force_next_flush(); + + PG_RETURN_VOID(); +} + + /* Reset all counters for the current database */ Datum pg_stat_reset(PG_FUNCTION_ARGS) -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0015-pgstat-utilize-pg_stat_force_next_flush-to-simpl.patch" From 0e8c0ccb795150b96b03ebd40c2383b8807694df Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sun, 27 Mar 2022 21:56:57 -0700 Subject: [PATCH v68 15/31] pgstat: utilize pg_stat_force_next_flush() to simplify tests. Author: Andres Freund Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- contrib/test_decoding/expected/stats.out | 71 ++----------- contrib/test_decoding/sql/stats.sql | 59 ++--------- src/test/regress/expected/brin.out | 22 ++++ src/test/regress/expected/stats.out | 123 +---------------------- src/test/regress/parallel_schedule | 5 +- src/test/regress/sql/brin.sql | 19 ++++ src/test/regress/sql/stats.sql | 119 +--------------------- 7 files changed, 64 insertions(+), 354 deletions(-) diff --git a/contrib/test_decoding/expected/stats.out b/contrib/test_decoding/expected/stats.out index a10f8214ce2..5dd1926176c 100644 --- a/contrib/test_decoding/expected/stats.out +++ b/contrib/test_decoding/expected/stats.out @@ -7,50 +7,6 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot_stats', ' (1 row) CREATE TABLE stats_test(data text); --- function to wait for counters to advance -CREATE FUNCTION wait_for_decode_stats(check_reset bool, check_spill_txns bool) RETURNS void AS $$ -DECLARE - start_time timestamptz := clock_timestamp(); - updated bool; -BEGIN - -- we don't want to wait forever; loop will exit after 30 seconds - FOR i IN 1 .. 300 LOOP - - IF check_spill_txns THEN - - -- check to see if all updates have been reset/updated - SELECT CASE WHEN check_reset THEN (spill_txns = 0) - ELSE (spill_txns > 0) - END - INTO updated - FROM pg_stat_replication_slots WHERE slot_name='regression_slot_stats'; - - ELSE - - -- check to see if all updates have been reset/updated - SELECT CASE WHEN check_reset THEN (total_txns = 0) - ELSE (total_txns > 0) - END - INTO updated - FROM pg_stat_replication_slots WHERE slot_name='regression_slot_stats'; - - END IF; - - exit WHEN updated; - - -- wait a little - perform pg_sleep_for('100 milliseconds'); - - -- reset stats snapshot so we can test again - perform pg_stat_clear_snapshot(); - - END LOOP; - - -- report time waited in postmaster log (where it won't change test output) - RAISE LOG 'wait_for_decode_stats delayed % seconds', - extract(epoch from clock_timestamp() - start_time); -END -$$ LANGUAGE plpgsql; -- non-spilled xact SET logical_decoding_work_mem to '64MB'; INSERT INTO stats_test values(1); @@ -60,9 +16,9 @@ SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot_stats', NULL, 3 (1 row) -SELECT wait_for_decode_stats(false, false); - wait_for_decode_stats ------------------------ +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- (1 row) @@ -73,19 +29,13 @@ SELECT slot_name, spill_txns = 0 AS spill_txns, spill_count = 0 AS spill_count, (1 row) RESET logical_decoding_work_mem; --- reset the slot stats, and wait for stats collector's total txn to reset +-- reset the slot stats SELECT pg_stat_reset_replication_slot('regression_slot_stats'); pg_stat_reset_replication_slot -------------------------------- (1 row) -SELECT wait_for_decode_stats(true, false); - wait_for_decode_stats ------------------------ - -(1 row) - SELECT slot_name, spill_txns, spill_count, total_txns, total_bytes FROM pg_stat_replication_slots; slot_name | spill_txns | spill_count | total_txns | total_bytes -----------------------+------------+-------------+------------+------------- @@ -102,12 +52,12 @@ SELECT count(*) FROM pg_logical_slot_peek_changes('regression_slot_stats', NULL, 5002 (1 row) --- Check stats, wait for the stats collector to update. We can't test the --- exact stats count as that can vary if any background transaction (say by --- autovacuum) happens in parallel to the main transaction. -SELECT wait_for_decode_stats(false, true); - wait_for_decode_stats ------------------------ +-- Check stats. We can't test the exact stats count as that can vary if any +-- background transaction (say by autovacuum) happens in parallel to the main +-- transaction. +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- (1 row) @@ -134,7 +84,6 @@ SELECT slot_name FROM pg_stat_replication_slots; (1 row) COMMIT; -DROP FUNCTION wait_for_decode_stats(bool, bool); DROP TABLE stats_test; SELECT pg_drop_replication_slot('regression_slot_stats'); pg_drop_replication_slot diff --git a/contrib/test_decoding/sql/stats.sql b/contrib/test_decoding/sql/stats.sql index 08616a2f965..464b9cd1fe3 100644 --- a/contrib/test_decoding/sql/stats.sql +++ b/contrib/test_decoding/sql/stats.sql @@ -5,62 +5,16 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot_stats', ' CREATE TABLE stats_test(data text); --- function to wait for counters to advance -CREATE FUNCTION wait_for_decode_stats(check_reset bool, check_spill_txns bool) RETURNS void AS $$ -DECLARE - start_time timestamptz := clock_timestamp(); - updated bool; -BEGIN - -- we don't want to wait forever; loop will exit after 30 seconds - FOR i IN 1 .. 300 LOOP - - IF check_spill_txns THEN - - -- check to see if all updates have been reset/updated - SELECT CASE WHEN check_reset THEN (spill_txns = 0) - ELSE (spill_txns > 0) - END - INTO updated - FROM pg_stat_replication_slots WHERE slot_name='regression_slot_stats'; - - ELSE - - -- check to see if all updates have been reset/updated - SELECT CASE WHEN check_reset THEN (total_txns = 0) - ELSE (total_txns > 0) - END - INTO updated - FROM pg_stat_replication_slots WHERE slot_name='regression_slot_stats'; - - END IF; - - exit WHEN updated; - - -- wait a little - perform pg_sleep_for('100 milliseconds'); - - -- reset stats snapshot so we can test again - perform pg_stat_clear_snapshot(); - - END LOOP; - - -- report time waited in postmaster log (where it won't change test output) - RAISE LOG 'wait_for_decode_stats delayed % seconds', - extract(epoch from clock_timestamp() - start_time); -END -$$ LANGUAGE plpgsql; - -- non-spilled xact SET logical_decoding_work_mem to '64MB'; INSERT INTO stats_test values(1); SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot_stats', NULL, NULL, 'skip-empty-xacts', '1'); -SELECT wait_for_decode_stats(false, false); +SELECT pg_stat_force_next_flush(); SELECT slot_name, spill_txns = 0 AS spill_txns, spill_count = 0 AS spill_count, total_txns > 0 AS total_txns, total_bytes > 0 AS total_bytes FROM pg_stat_replication_slots; RESET logical_decoding_work_mem; --- reset the slot stats, and wait for stats collector's total txn to reset +-- reset the slot stats SELECT pg_stat_reset_replication_slot('regression_slot_stats'); -SELECT wait_for_decode_stats(true, false); SELECT slot_name, spill_txns, spill_count, total_txns, total_bytes FROM pg_stat_replication_slots; -- spilling the xact @@ -69,10 +23,10 @@ INSERT INTO stats_test SELECT 'serialize-topbig--1:'||g.i FROM generate_series(1 COMMIT; SELECT count(*) FROM pg_logical_slot_peek_changes('regression_slot_stats', NULL, NULL, 'skip-empty-xacts', '1'); --- Check stats, wait for the stats collector to update. We can't test the --- exact stats count as that can vary if any background transaction (say by --- autovacuum) happens in parallel to the main transaction. -SELECT wait_for_decode_stats(false, true); +-- Check stats. We can't test the exact stats count as that can vary if any +-- background transaction (say by autovacuum) happens in parallel to the main +-- transaction. +SELECT pg_stat_force_next_flush(); SELECT slot_name, spill_txns > 0 AS spill_txns, spill_count > 0 AS spill_count FROM pg_stat_replication_slots; -- Ensure stats can be repeatedly accessed using the same stats snapshot. See @@ -83,6 +37,5 @@ SELECT slot_name FROM pg_stat_replication_slots; SELECT slot_name FROM pg_stat_replication_slots; COMMIT; -DROP FUNCTION wait_for_decode_stats(bool, bool); DROP TABLE stats_test; SELECT pg_drop_replication_slot('regression_slot_stats'); diff --git a/src/test/regress/expected/brin.out b/src/test/regress/expected/brin.out index 2d03d8e134e..ed7879f5831 100644 --- a/src/test/regress/expected/brin.out +++ b/src/test/regress/expected/brin.out @@ -603,3 +603,25 @@ SELECT COUNT(*) FROM brin_hot_2 WHERE a = 2 AND b = 100; 1 (1 row) +-- test BRIN index doesn't block HOT update +CREATE TABLE brin_hot ( + id integer PRIMARY KEY, + val integer NOT NULL +) WITH (autovacuum_enabled = off, fillfactor = 70); +INSERT INTO brin_hot SELECT *, 0 FROM generate_series(1, 235); +CREATE INDEX val_brin ON brin_hot using brin(val); +UPDATE brin_hot SET val = -3 WHERE id = 42; +-- ensure pending stats are flushed +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT pg_stat_get_tuples_hot_updated('brin_hot'::regclass::oid); + pg_stat_get_tuples_hot_updated +-------------------------------- + 1 +(1 row) + +DROP TABLE brin_hot; diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out index 2f824258cd6..1a9b6d9ec9b 100644 --- a/src/test/regress/expected/stats.out +++ b/src/test/regress/expected/stats.out @@ -28,63 +28,6 @@ SELECT t.seq_scan, t.seq_tup_read, t.idx_scan, t.idx_tup_fetch, pg_catalog.pg_statio_user_tables AS b WHERE t.relname='tenk2' AND b.relname='tenk2'; COMMIT; --- function to wait for counters to advance -create function wait_for_stats() returns void as $$ -declare - start_time timestamptz := clock_timestamp(); - updated1 bool; - updated2 bool; - updated3 bool; - updated4 bool; -begin - SET LOCAL stats_fetch_consistency = snapshot; - - -- We don't want to wait forever. No timeout suffices if the OS drops our - -- stats traffic because an earlier test file left a full UDP buffer. - -- Hence, don't use PG_TEST_TIMEOUT_DEFAULT, which may be large for - -- can't-happen timeouts. Exit after 30 seconds. - for i in 1 .. 300 loop - - -- With parallel query, the seqscan and indexscan on tenk2 might be done - -- in parallel worker processes, which will send their stats counters - -- asynchronously to what our own session does. So we must check for - -- those counts to be registered separately from the update counts. - - -- check to see if seqscan has been sensed - SELECT (st.seq_scan >= pr.seq_scan + 1) INTO updated1 - FROM pg_stat_user_tables AS st, pg_class AS cl, prevstats AS pr - WHERE st.relname='tenk2' AND cl.relname='tenk2'; - - -- check to see if indexscan has been sensed - SELECT (st.idx_scan >= pr.idx_scan + 1) INTO updated2 - FROM pg_stat_user_tables AS st, pg_class AS cl, prevstats AS pr - WHERE st.relname='tenk2' AND cl.relname='tenk2'; - - -- check to see if all updates have been sensed - SELECT (n_tup_ins > 0) INTO updated3 - FROM pg_stat_user_tables WHERE relname='trunc_stats_test4'; - - -- We must also check explicitly that pg_stat_get_snapshot_timestamp has - -- advanced, because that comes from the global stats file which might - -- be older than the per-DB stats file we got the other values from. - SELECT (pr.snap_ts < pg_stat_get_snapshot_timestamp()) INTO updated4 - FROM prevstats AS pr; - - exit when updated1 and updated2 and updated3 and updated4; - - -- wait a little - perform pg_sleep_for('100 milliseconds'); - - -- reset stats snapshot so we can test again - perform pg_stat_clear_snapshot(); - - end loop; - - -- report time waited in postmaster log (where it won't change test output) - raise log 'wait_for_stats delayed % seconds', - extract(epoch from clock_timestamp() - start_time); -end -$$ language plpgsql; -- test effects of TRUNCATE on n_live_tup/n_dead_tup counters CREATE TABLE trunc_stats_test(id serial); CREATE TABLE trunc_stats_test1(id serial, stuff text); @@ -153,17 +96,10 @@ SELECT count(*) FROM tenk2 WHERE unique1 = 1; (1 row) RESET enable_bitmapscan; --- We can't just call wait_for_stats() at this point, because we only --- transmit stats when the session goes idle, and we probably didn't --- transmit the last couple of counts yet thanks to the rate-limiting logic --- in pgstat_report_stat(). But instead of waiting for the rate limiter's --- timeout to elapse, let's just start a new session. The old one will --- then send its stats before dying. -\c - --- wait for stats collector to update -SELECT wait_for_stats(); - wait_for_stats ----------------- +-- ensure pending stats are flushed +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- (1 row) @@ -212,57 +148,6 @@ FROM prevstats AS pr; COMMIT; DROP TABLE trunc_stats_test, trunc_stats_test1, trunc_stats_test2, trunc_stats_test3, trunc_stats_test4; DROP TABLE prevstats; --- test BRIN index doesn't block HOT update - we include this test here, as it --- relies on statistics collector and so it may occasionally fail, especially --- on slower systems -CREATE TABLE brin_hot ( - id integer PRIMARY KEY, - val integer NOT NULL -) WITH (autovacuum_enabled = off, fillfactor = 70); -INSERT INTO brin_hot SELECT *, 0 FROM generate_series(1, 235); -CREATE INDEX val_brin ON brin_hot using brin(val); -CREATE FUNCTION wait_for_hot_stats() RETURNS void AS $$ -DECLARE - start_time timestamptz := clock_timestamp(); - updated bool; -BEGIN - -- we don't want to wait forever; loop will exit after 30 seconds - FOR i IN 1 .. 300 LOOP - SELECT (pg_stat_get_tuples_hot_updated('brin_hot'::regclass::oid) > 0) INTO updated; - EXIT WHEN updated; - - -- wait a little - PERFORM pg_sleep_for('100 milliseconds'); - -- reset stats snapshot so we can test again - PERFORM pg_stat_clear_snapshot(); - END LOOP; - -- report time waited in postmaster log (where it won't change test output) - RAISE log 'wait_for_hot_stats delayed % seconds', - EXTRACT(epoch FROM clock_timestamp() - start_time); -END -$$ LANGUAGE plpgsql; -UPDATE brin_hot SET val = -3 WHERE id = 42; --- We can't just call wait_for_hot_stats() at this point, because we only --- transmit stats when the session goes idle, and we probably didn't --- transmit the last couple of counts yet thanks to the rate-limiting logic --- in pgstat_report_stat(). But instead of waiting for the rate limiter's --- timeout to elapse, let's just start a new session. The old one will --- then send its stats before dying. -\c - -SELECT wait_for_hot_stats(); - wait_for_hot_stats --------------------- - -(1 row) - -SELECT pg_stat_get_tuples_hot_updated('brin_hot'::regclass::oid); - pg_stat_get_tuples_hot_updated --------------------------------- - 1 -(1 row) - -DROP TABLE brin_hot; -DROP FUNCTION wait_for_hot_stats(); -- ensure that stats accessors handle NULL input correctly SELECT pg_stat_get_replication_slot(NULL); pg_stat_get_replication_slot diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 5030d19c03e..1087b2c14f4 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -124,7 +124,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression memoize +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression memoize stats # event_trigger cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage @@ -132,6 +132,3 @@ test: event_trigger oidjoins # this test also uses event triggers, so likewise run it by itself test: fast_default - -# run stats by itself because its delay may be insufficient under heavy load -test: stats diff --git a/src/test/regress/sql/brin.sql b/src/test/regress/sql/brin.sql index e12f3a0df98..920e0532494 100644 --- a/src/test/regress/sql/brin.sql +++ b/src/test/regress/sql/brin.sql @@ -526,3 +526,22 @@ SET enable_seqscan = off; EXPLAIN (COSTS OFF) SELECT * FROM brin_hot_2 WHERE a = 2 AND b = 100; SELECT COUNT(*) FROM brin_hot_2 WHERE a = 2 AND b = 100; + + +-- test BRIN index doesn't block HOT update +CREATE TABLE brin_hot ( + id integer PRIMARY KEY, + val integer NOT NULL +) WITH (autovacuum_enabled = off, fillfactor = 70); + +INSERT INTO brin_hot SELECT *, 0 FROM generate_series(1, 235); +CREATE INDEX val_brin ON brin_hot using brin(val); + +UPDATE brin_hot SET val = -3 WHERE id = 42; + +-- ensure pending stats are flushed +SELECT pg_stat_force_next_flush(); + +SELECT pg_stat_get_tuples_hot_updated('brin_hot'::regclass::oid); + +DROP TABLE brin_hot; diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql index 4f33ea6a7c8..d49e0aeef85 100644 --- a/src/test/regress/sql/stats.sql +++ b/src/test/regress/sql/stats.sql @@ -27,64 +27,6 @@ SELECT t.seq_scan, t.seq_tup_read, t.idx_scan, t.idx_tup_fetch, WHERE t.relname='tenk2' AND b.relname='tenk2'; COMMIT; --- function to wait for counters to advance -create function wait_for_stats() returns void as $$ -declare - start_time timestamptz := clock_timestamp(); - updated1 bool; - updated2 bool; - updated3 bool; - updated4 bool; -begin - SET LOCAL stats_fetch_consistency = snapshot; - - -- We don't want to wait forever. No timeout suffices if the OS drops our - -- stats traffic because an earlier test file left a full UDP buffer. - -- Hence, don't use PG_TEST_TIMEOUT_DEFAULT, which may be large for - -- can't-happen timeouts. Exit after 30 seconds. - for i in 1 .. 300 loop - - -- With parallel query, the seqscan and indexscan on tenk2 might be done - -- in parallel worker processes, which will send their stats counters - -- asynchronously to what our own session does. So we must check for - -- those counts to be registered separately from the update counts. - - -- check to see if seqscan has been sensed - SELECT (st.seq_scan >= pr.seq_scan + 1) INTO updated1 - FROM pg_stat_user_tables AS st, pg_class AS cl, prevstats AS pr - WHERE st.relname='tenk2' AND cl.relname='tenk2'; - - -- check to see if indexscan has been sensed - SELECT (st.idx_scan >= pr.idx_scan + 1) INTO updated2 - FROM pg_stat_user_tables AS st, pg_class AS cl, prevstats AS pr - WHERE st.relname='tenk2' AND cl.relname='tenk2'; - - -- check to see if all updates have been sensed - SELECT (n_tup_ins > 0) INTO updated3 - FROM pg_stat_user_tables WHERE relname='trunc_stats_test4'; - - -- We must also check explicitly that pg_stat_get_snapshot_timestamp has - -- advanced, because that comes from the global stats file which might - -- be older than the per-DB stats file we got the other values from. - SELECT (pr.snap_ts < pg_stat_get_snapshot_timestamp()) INTO updated4 - FROM prevstats AS pr; - - exit when updated1 and updated2 and updated3 and updated4; - - -- wait a little - perform pg_sleep_for('100 milliseconds'); - - -- reset stats snapshot so we can test again - perform pg_stat_clear_snapshot(); - - end loop; - - -- report time waited in postmaster log (where it won't change test output) - raise log 'wait_for_stats delayed % seconds', - extract(epoch from clock_timestamp() - start_time); -end -$$ language plpgsql; - -- test effects of TRUNCATE on n_live_tup/n_dead_tup counters CREATE TABLE trunc_stats_test(id serial); CREATE TABLE trunc_stats_test1(id serial, stuff text); @@ -151,16 +93,8 @@ SET enable_bitmapscan TO off; SELECT count(*) FROM tenk2 WHERE unique1 = 1; RESET enable_bitmapscan; --- We can't just call wait_for_stats() at this point, because we only --- transmit stats when the session goes idle, and we probably didn't --- transmit the last couple of counts yet thanks to the rate-limiting logic --- in pgstat_report_stat(). But instead of waiting for the rate limiter's --- timeout to elapse, let's just start a new session. The old one will --- then send its stats before dying. -\c - - --- wait for stats collector to update -SELECT wait_for_stats(); +-- ensure pending stats are flushed +SELECT pg_stat_force_next_flush(); -- check effects BEGIN; @@ -190,55 +124,6 @@ COMMIT; DROP TABLE trunc_stats_test, trunc_stats_test1, trunc_stats_test2, trunc_stats_test3, trunc_stats_test4; DROP TABLE prevstats; --- test BRIN index doesn't block HOT update - we include this test here, as it --- relies on statistics collector and so it may occasionally fail, especially --- on slower systems -CREATE TABLE brin_hot ( - id integer PRIMARY KEY, - val integer NOT NULL -) WITH (autovacuum_enabled = off, fillfactor = 70); - -INSERT INTO brin_hot SELECT *, 0 FROM generate_series(1, 235); -CREATE INDEX val_brin ON brin_hot using brin(val); - -CREATE FUNCTION wait_for_hot_stats() RETURNS void AS $$ -DECLARE - start_time timestamptz := clock_timestamp(); - updated bool; -BEGIN - -- we don't want to wait forever; loop will exit after 30 seconds - FOR i IN 1 .. 300 LOOP - SELECT (pg_stat_get_tuples_hot_updated('brin_hot'::regclass::oid) > 0) INTO updated; - EXIT WHEN updated; - - -- wait a little - PERFORM pg_sleep_for('100 milliseconds'); - -- reset stats snapshot so we can test again - PERFORM pg_stat_clear_snapshot(); - END LOOP; - -- report time waited in postmaster log (where it won't change test output) - RAISE log 'wait_for_hot_stats delayed % seconds', - EXTRACT(epoch FROM clock_timestamp() - start_time); -END -$$ LANGUAGE plpgsql; - -UPDATE brin_hot SET val = -3 WHERE id = 42; - --- We can't just call wait_for_hot_stats() at this point, because we only --- transmit stats when the session goes idle, and we probably didn't --- transmit the last couple of counts yet thanks to the rate-limiting logic --- in pgstat_report_stat(). But instead of waiting for the rate limiter's --- timeout to elapse, let's just start a new session. The old one will --- then send its stats before dying. -\c - - -SELECT wait_for_hot_stats(); -SELECT pg_stat_get_tuples_hot_updated('brin_hot'::regclass::oid); - -DROP TABLE brin_hot; -DROP FUNCTION wait_for_hot_stats(); - - -- ensure that stats accessors handle NULL input correctly SELECT pg_stat_get_replication_slot(NULL); SELECT pg_stat_get_subscription_stats(NULL); -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0016-pgstat-add-pg_stat_exists_stat-for-easier-testin.patch" From fd87c2e8e8bf7bdfec3ea2a5585fb2d57b41b3eb Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sat, 2 Apr 2022 14:21:58 -0700 Subject: [PATCH v68 16/31] pgstat: add pg_stat_exists_stat() for easier testing. Useful for tests. Author: Melanie Plageman --- src/include/catalog/pg_proc.dat | 6 ++++++ src/include/pgstat.h | 2 ++ src/backend/catalog/system_functions.sql | 2 ++ src/backend/postmaster/pgstat.c | 9 +++++++++ src/backend/utils/adt/pgstatfuncs.c | 17 +++++++++++++++++ 5 files changed, 36 insertions(+) diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 12526c599e0..948e6504da0 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -5376,6 +5376,12 @@ proargmodes => '{i,o,o,o,o,o,o,o,o,o,o}', proargnames => '{slot_name,slot_name,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes,total_txns,total_bytes,stats_reset}', prosrc => 'pg_stat_get_replication_slot' }, + +{ oid => '8384', descr => 'statistics: check if a stats object exists', + proname => 'pg_stat_exists_stat', provolatile => 'v', proparallel => 'r', + prorettype => 'bool', proargtypes => 'text oid oid', + prosrc => 'pg_stat_exists_stat' }, + { oid => '8523', descr => 'statistics: information about subscription stats', proname => 'pg_stat_get_subscription_stats', provolatile => 's', proparallel => 'r', diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 3481ba5e262..fae3186c8eb 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -438,6 +438,8 @@ extern TimestampTz pgstat_get_stat_snapshot_timestamp(bool *have_snapshot); /* helpers */ extern PgStat_Kind pgstat_kind_from_str(char *kind_str); +extern bool pgstat_exists_entry(PgStat_Kind kind, Oid dboid, Oid objoid); + /* * Functions in pgstat_archiver.c diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql index 81bac6f5812..07386f97f95 100644 --- a/src/backend/catalog/system_functions.sql +++ b/src/backend/catalog/system_functions.sql @@ -639,6 +639,8 @@ REVOKE EXECUTE ON FUNCTION pg_stat_reset_single_function_counters(oid) FROM publ REVOKE EXECUTE ON FUNCTION pg_stat_reset_replication_slot(text) FROM public; +REVOKE EXECUTE ON FUNCTION pg_stat_exists_stat(text, oid, oid) FROM public; + REVOKE EXECUTE ON FUNCTION pg_stat_reset_subscription_stats(oid) FROM public; REVOKE EXECUTE ON FUNCTION lo_import(text) FROM public; diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index 0a0d64be829..8d619d114b2 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/postmaster/pgstat.c @@ -835,6 +835,15 @@ pgstat_get_stat_snapshot_timestamp(bool *have_snapshot) return 0; } +bool +pgstat_exists_entry(PgStat_Kind kind, Oid dboid, Oid objoid) +{ + if (pgstat_kind_info_for(kind)->fixed_amount) + return true; + + return pgstat_get_entry_ref(kind, dboid, objoid, false, NULL) != NULL; +} + /* * Ensure snapshot for a kind of global stats exist. * diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index caaf0c5a672..2660bd1eb07 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -2395,3 +2395,20 @@ pg_stat_get_subscription_stats(PG_FUNCTION_ARGS) /* Returns the record as Datum */ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); } + +/* + * Checks for presence of stats for object with provided kind, database oid, + * object oid. + * + * This is useful for regression tests, but not much more. + */ +Datum +pg_stat_exists_stat(PG_FUNCTION_ARGS) +{ + char *stats_type = text_to_cstring(PG_GETARG_TEXT_P(0)); + Oid dboid = PG_GETARG_OID(1); + Oid objoid = PG_GETARG_OID(2); + PgStat_Kind kind = pgstat_kind_from_str(stats_type); + + PG_RETURN_BOOL(pgstat_exists_entry(kind, dboid, objoid)); +} -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0017-pgstat-wip-only-reset-pgstat-data-after-crash-re.patch" From ad4acc0477860b1f2c45808c1996e0d22da6a821 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Wed, 16 Mar 2022 23:16:10 -0700 Subject: [PATCH v68 17/31] pgstat: wip: only reset pgstat data after crash recovery. Previously we always reset it when performing WAL replay, even if starting in DB_SHUTDOWNED_IN_RECOVERY. --- src/backend/access/transam/xlog.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index be2cd1a3ea5..c95937f9857 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -5133,6 +5133,20 @@ StartupXLOG(void) */ restoreTwoPhaseData(); + /* + * When starting with crash recovery, reset pgstat data - it might not be + * valid. Otherwise restore pgstat data. It's safe to do this here, + * because postmaster will not yet have started any other processes + * + * TODO: With a bit of extra work we could just start with a pgstat file + * associated with the checkpoint redo location we're starting from. + */ + if (ControlFile->state == DB_SHUTDOWNED || + ControlFile->state == DB_SHUTDOWNED_IN_RECOVERY) + pgstat_restore_stats(); + else + pgstat_discard_stats(); + lastFullPageWrites = checkPoint.fullPageWrites; RedoRecPtr = XLogCtl->RedoRecPtr = XLogCtl->Insert.RedoRecPtr = checkPoint.redo; @@ -5207,14 +5221,6 @@ StartupXLOG(void) LocalMinRecoveryPointTLI = 0; } - /* - * Reset pgstat data, because it may be invalid after recovery. It's - * safe to do this here, because postmaster will not yet have started - * any other processes. NB: This basically just skips loading the data - * from disk, see pgstat_restore_stats() call in clean-startup path. - */ - pgstat_discard_stats(); - /* Check that the GUCs used to generate the WAL allow recovery */ CheckRequiredParameterValues(); @@ -5307,9 +5313,6 @@ StartupXLOG(void) else { performedWalRecovery = false; - - /* AFIXME: better location? */ - pgstat_restore_stats(); } /* -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0018-pgstat-test-transaction-behaviour-2PC-function-s.patch" From 2062927dfbfe233c93e8d211cdc5cdcdbc832f41 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Wed, 16 Mar 2022 17:11:28 -0700 Subject: [PATCH v68 18/31] pgstat: test: transaction behaviour, 2PC, function stats. Author: Andres Freund Author: Melanie Plageman Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- src/test/isolation/expected/stats.out | 3726 +++++++++++++++++++++++++ src/test/isolation/isolation_schedule | 1 + src/test/isolation/specs/stats.spec | 753 +++++ src/test/regress/expected/stats.out | 368 +++ src/test/regress/parallel_schedule | 3 + src/test/regress/sql/stats.sql | 158 ++ 6 files changed, 5009 insertions(+) create mode 100644 src/test/isolation/expected/stats.out create mode 100644 src/test/isolation/specs/stats.spec diff --git a/src/test/isolation/expected/stats.out b/src/test/isolation/expected/stats.out new file mode 100644 index 00000000000..9ff4201b66f --- /dev/null +++ b/src/test/isolation/expected/stats.out @@ -0,0 +1,3726 @@ +Parsed test spec with 2 sessions + +starting permutation: s1_track_funcs_none s1_func_stats s1_func_call s1_func_call s1_ff s1_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_none: SET track_functions = 'none'; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + + +starting permutation: s1_track_funcs_all s1_func_stats s1_func_call s1_func_call s1_ff s1_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 2|t |t +(1 row) + + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_func_stats s2_func_stats s1_func_call s2_func_call s1_func_call s2_func_call s2_func_call s1_ff s2_ff s1_func_stats s2_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 5|t |t +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 5|t |t +(1 row) + + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_func_stats s2_func_stats s1_func_call s1_ff s2_func_call s2_func_call s2_ff s1_func_stats s2_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 3|t |t +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 3|t |t +(1 row) + + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_func_stats s2_func_stats s1_begin s1_func_call s1_func_call s1_commit s1_ff s1_func_stats s2_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s1_begin: BEGIN; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_commit: COMMIT; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 2|t |t +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 2|t |t +(1 row) + + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_func_stats s2_func_stats s1_begin s1_func_call s2_func_call s1_func_drop s2_func_call s2_ff s2_func_stats s1_commit s1_ff s1_func_stats s2_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s1_begin: BEGIN; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s1_func_drop: DROP FUNCTION test_stat_func(); +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 2|t |t +(1 row) + +step s1_commit: COMMIT; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_func_stats s2_func_stats s1_begin s1_func_call s2_func_call s1_func_drop s2_func_call s2_ff s2_func_stats s1_rollback s1_ff s1_func_stats s2_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s1_begin: BEGIN; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s1_func_drop: DROP FUNCTION test_stat_func(); +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 2|t |t +(1 row) + +step s1_rollback: ROLLBACK; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 3|t |t +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 3|t |t +(1 row) + + +starting permutation: s1_track_funcs_all s2_track_funcs_all s2_func_call s2_ff s2_begin s2_func_call s1_func_drop s1_func_stats s2_commit s2_ff s1_func_stats s2_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_begin: BEGIN; +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s1_func_drop: DROP FUNCTION test_stat_func(); +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_commit: COMMIT; +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + + +starting permutation: s1_track_funcs_all s2_track_funcs_all s2_begin s2_func_call s1_func_drop s1_func_stats s2_commit s2_ff s1_func_stats s2_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s2_begin: BEGIN; +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s1_func_drop: DROP FUNCTION test_stat_func(); +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_commit: COMMIT; +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + + +starting permutation: s1_disable_debug_discard s2_disable_debug_discard s1_track_funcs_all s2_track_funcs_all s1_func_call s2_begin s2_func_call s1_func_drop s2_func_call s2_commit s2_ff s1_func_stats s2_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_disable_debug_discard: SET debug_discard_caches = 0; +step s2_disable_debug_discard: SET debug_discard_caches = 0; +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s2_begin: BEGIN; +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s1_func_drop: DROP FUNCTION test_stat_func(); +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_commit: COMMIT; +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + + +starting permutation: s1_disable_debug_discard s2_disable_debug_discard s1_track_funcs_all s2_track_funcs_none s1_func_call s2_begin s2_func_call s1_ff s1_func_stats s1_func_drop s2_track_funcs_none s1_func_stats s2_func_call s2_commit s2_ff s1_func_stats s2_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_disable_debug_discard: SET debug_discard_caches = 0; +step s2_disable_debug_discard: SET debug_discard_caches = 0; +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_none: SET track_functions = 'none'; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s2_begin: BEGIN; +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 1|t |t +(1 row) + +step s1_func_drop: DROP FUNCTION test_stat_func(); +step s2_track_funcs_none: SET track_functions = 'none'; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_commit: COMMIT; +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + + +starting permutation: s1_disable_debug_discard s2_disable_debug_discard s1_track_funcs_all s2_track_funcs_none s1_func_call s2_begin s2_func_call s1_ff s1_func_stats s1_func_drop s2_track_funcs_all s1_func_stats s2_func_call s2_commit s2_ff s1_func_stats s2_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_disable_debug_discard: SET debug_discard_caches = 0; +step s2_disable_debug_discard: SET debug_discard_caches = 0; +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_none: SET track_functions = 'none'; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s2_begin: BEGIN; +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 1|t |t +(1 row) + +step s1_func_drop: DROP FUNCTION test_stat_func(); +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_func_call: SELECT test_stat_func() +ERROR: function call to dropped function +step s2_commit: COMMIT; +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_func_call s2_func_call s2_func_call2 s1_ff s2_ff s1_func_stats s2_func_call s2_func_call2 s2_ff s1_func_stats s1_func_stats2 s1_func_stats s1_func_stats_reset s1_func_stats s1_func_stats2 s1_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_func_call2: SELECT test_stat_func2() +test_stat_func2 +--------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 2|t |t +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_func_call2: SELECT test_stat_func2() +test_stat_func2 +--------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 3|t |t +(1 row) + +step s1_func_stats2: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func2' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +---------------+--------------------------+----------------+--------------- +test_stat_func2| 2|t |t +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 3|t |t +(1 row) + +step s1_func_stats_reset: SELECT pg_stat_reset_single_function_counters('test_stat_func'::regproc); +pg_stat_reset_single_function_counters +-------------------------------------- + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 0|f |f +(1 row) + +step s1_func_stats2: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func2' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +---------------+--------------------------+----------------+--------------- +test_stat_func2| 2|t |t +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 0|f |f +(1 row) + + +starting permutation: s1_func_stats_nonexistent s1_func_stats_reset_nonexistent s1_func_stats_nonexistent +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats_nonexistent: + SELECT pg_stat_get_function_calls(12000); + +pg_stat_get_function_calls +-------------------------- + +(1 row) + +step s1_func_stats_reset_nonexistent: SELECT pg_stat_reset_single_function_counters(12000); +pg_stat_reset_single_function_counters +-------------------------------------- + +(1 row) + +step s1_func_stats_nonexistent: + SELECT pg_stat_get_function_calls(12000); + +pg_stat_get_function_calls +-------------------------- + +(1 row) + + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_func_call s2_func_call s2_func_call2 s1_ff s2_ff s1_func_stats s1_func_stats2 s1_func_stats s1_reset s1_func_stats s1_func_stats2 s1_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_func_call2: SELECT test_stat_func2() +test_stat_func2 +--------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 2|t |t +(1 row) + +step s1_func_stats2: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func2' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +---------------+--------------------------+----------------+--------------- +test_stat_func2| 1|t |t +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 2|t |t +(1 row) + +step s1_reset: SELECT pg_stat_reset(); +pg_stat_reset +------------- + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 0|f |f +(1 row) + +step s1_func_stats2: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func2' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +---------------+--------------------------+----------------+--------------- +test_stat_func2| 0|f |f +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 0|f |f +(1 row) + + +starting permutation: s1_track_funcs_all s1_fetch_consistency_none s1_func_call s1_ff s1_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s1_fetch_consistency_none: SET stats_fetch_consistency = 'none'; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 1|t |t +(1 row) + + +starting permutation: s1_track_funcs_all s1_fetch_consistency_cache s1_func_call s1_ff s1_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s1_fetch_consistency_cache: SET stats_fetch_consistency = 'cache'; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 1|t |t +(1 row) + + +starting permutation: s1_track_funcs_all s1_fetch_consistency_snapshot s1_func_call s1_ff s1_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s1_fetch_consistency_snapshot: SET stats_fetch_consistency = 'snapshot'; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 1|t |t +(1 row) + + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_fetch_consistency_none s2_func_call s2_ff s1_begin s1_func_stats s2_func_call s2_ff s1_func_stats s1_commit +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_fetch_consistency_none: SET stats_fetch_consistency = 'none'; +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_begin: BEGIN; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 1|t |t +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 2|t |t +(1 row) + +step s1_commit: COMMIT; + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_fetch_consistency_cache s2_func_call s2_func_call2 s2_ff s1_begin s1_func_stats s2_func_call s2_func_call2 s2_ff s1_func_stats s1_func_stats2 s1_commit +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_fetch_consistency_cache: SET stats_fetch_consistency = 'cache'; +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_func_call2: SELECT test_stat_func2() +test_stat_func2 +--------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_begin: BEGIN; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 1|t |t +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_func_call2: SELECT test_stat_func2() +test_stat_func2 +--------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 1|t |t +(1 row) + +step s1_func_stats2: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func2' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +---------------+--------------------------+----------------+--------------- +test_stat_func2| 2|t |t +(1 row) + +step s1_commit: COMMIT; + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_fetch_consistency_snapshot s2_func_call s2_func_call2 s2_ff s1_begin s1_func_stats s2_func_call s2_func_call2 s2_ff s1_func_stats s1_func_stats2 s1_commit +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_fetch_consistency_snapshot: SET stats_fetch_consistency = 'snapshot'; +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_func_call2: SELECT test_stat_func2() +test_stat_func2 +--------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_begin: BEGIN; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 1|t |t +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_func_call2: SELECT test_stat_func2() +test_stat_func2 +--------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 1|t |t +(1 row) + +step s1_func_stats2: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func2' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +---------------+--------------------------+----------------+--------------- +test_stat_func2| 1|t |t +(1 row) + +step s1_commit: COMMIT; + +starting permutation: s1_fetch_consistency_none s1_begin s1_func_stats_nonexistent s1_func_stats_nonexistent s1_commit +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_fetch_consistency_none: SET stats_fetch_consistency = 'none'; +step s1_begin: BEGIN; +step s1_func_stats_nonexistent: + SELECT pg_stat_get_function_calls(12000); + +pg_stat_get_function_calls +-------------------------- + +(1 row) + +step s1_func_stats_nonexistent: + SELECT pg_stat_get_function_calls(12000); + +pg_stat_get_function_calls +-------------------------- + +(1 row) + +step s1_commit: COMMIT; + +starting permutation: s1_fetch_consistency_cache s1_begin s1_func_stats_nonexistent s1_func_stats_nonexistent s1_commit +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_fetch_consistency_cache: SET stats_fetch_consistency = 'cache'; +step s1_begin: BEGIN; +step s1_func_stats_nonexistent: + SELECT pg_stat_get_function_calls(12000); + +pg_stat_get_function_calls +-------------------------- + +(1 row) + +step s1_func_stats_nonexistent: + SELECT pg_stat_get_function_calls(12000); + +pg_stat_get_function_calls +-------------------------- + +(1 row) + +step s1_commit: COMMIT; + +starting permutation: s1_fetch_consistency_snapshot s1_begin s1_func_stats_nonexistent s1_func_stats_nonexistent s1_commit +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_fetch_consistency_snapshot: SET stats_fetch_consistency = 'snapshot'; +step s1_begin: BEGIN; +step s1_func_stats_nonexistent: + SELECT pg_stat_get_function_calls(12000); + +pg_stat_get_function_calls +-------------------------- + +(1 row) + +step s1_func_stats_nonexistent: + SELECT pg_stat_get_function_calls(12000); + +pg_stat_get_function_calls +-------------------------- + +(1 row) + +step s1_commit: COMMIT; + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_begin s1_func_call s2_func_call s1_func_drop s2_func_call s2_ff s1_prepare_a s2_func_call s2_ff s1_func_call s1_ff s1_func_stats s1_commit_prepared_a s1_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_begin: BEGIN; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s1_func_drop: DROP FUNCTION test_stat_func(); +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 5|t |t +(1 row) + +step s1_commit_prepared_a: COMMIT PREPARED 'a'; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_begin s1_func_call s2_func_call s1_func_drop s2_func_call s2_ff s1_prepare_a s2_func_call s2_ff s1_func_call s1_ff s1_func_stats s1_rollback_prepared_a s1_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_begin: BEGIN; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s1_func_drop: DROP FUNCTION test_stat_func(); +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 5|t |t +(1 row) + +step s1_rollback_prepared_a: ROLLBACK PREPARED 'a'; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 5|t |t +(1 row) + + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_begin s1_func_call s2_func_call s1_func_drop s2_func_call s2_ff s1_prepare_a s2_func_call s2_ff s1_func_call s1_ff s1_func_stats s2_commit_prepared_a s1_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_begin: BEGIN; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s1_func_drop: DROP FUNCTION test_stat_func(); +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 5|t |t +(1 row) + +step s2_commit_prepared_a: COMMIT PREPARED 'a'; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + + +starting permutation: s1_track_funcs_all s2_track_funcs_all s1_begin s1_func_call s2_func_call s1_func_drop s2_func_call s2_ff s1_prepare_a s2_func_call s2_ff s1_func_call s1_ff s1_func_stats s2_rollback_prepared_a s1_func_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_funcs_all: SET track_functions = 'all'; +step s2_track_funcs_all: SET track_functions = 'all'; +step s1_begin: BEGIN; +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s1_func_drop: DROP FUNCTION test_stat_func(); +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_call: SELECT test_stat_func(); +test_stat_func +-------------- + +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 5|t |t +(1 row) + +step s2_rollback_prepared_a: ROLLBACK PREPARED 'a'; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 5|t |t +(1 row) + + +starting permutation: s1_table_select s1_table_insert s2_table_select s2_table_update_k1 s1_ff s2_table_update_k1 s1_table_drop s2_ff s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +(1 row) + +step s1_table_insert: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1), ('k2', 1), ('k3', 1); +step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +k1 | 1 +k2 | 1 +k3 | 1 +(4 rows) + +step s2_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_drop: DROP TABLE test_stat_tab; +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 0| 0| 0| 0| 0| 0| 0| 0 +(1 row) + + +starting permutation: s1_table_select s1_table_insert s2_table_select s2_table_update_k1 s2_table_update_k1 s1_table_drop s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +(1 row) + +step s1_table_insert: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1), ('k2', 1), ('k3', 1); +step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +k1 | 1 +k2 | 1 +k3 | 1 +(4 rows) + +step s2_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s2_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_drop: DROP TABLE test_stat_tab; +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 0| 0| 0| 0| 0| 0| 0| 0 +(1 row) + + +starting permutation: s1_track_counts_off s1_table_stats s1_track_counts_on +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_counts_off: SET track_counts = off; +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 0| 0| 1| 0| 0| 1| 0| 0 +(1 row) + +step s1_track_counts_on: SET track_counts = on; + +starting permutation: s1_table_select s1_track_counts_off s1_ff s1_table_stats s1_track_counts_on +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +(1 row) + +step s1_track_counts_off: SET track_counts = off; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 1| 1| 1| 0| 0| 1| 0| 0 +(1 row) + +step s1_track_counts_on: SET track_counts = on; + +starting permutation: s1_table_select s1_ff s1_track_counts_off s1_table_stats s1_track_counts_on +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_counts_off: SET track_counts = off; +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 1| 1| 1| 0| 0| 1| 0| 0 +(1 row) + +step s1_track_counts_on: SET track_counts = on; + +starting permutation: s1_track_counts_off s1_table_select s1_table_insert_k1 s1_table_update_k1 s2_table_select s1_track_counts_on s1_ff s2_ff s1_table_stats s1_table_select s1_table_update_k1 s1_ff s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_track_counts_off: SET track_counts = off; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +(1 row) + +step s1_table_insert_k1: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +k1 | 2 +(2 rows) + +step s1_track_counts_on: SET track_counts = on; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 1| 2| 1| 0| 0| 1| 0| 0 +(1 row) + +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +k1 | 2 +(2 rows) + +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 3| 6| 1| 1| 0| 1| 1| 0 +(1 row) + + +starting permutation: s1_table_select s1_table_insert_k1 s1_table_delete_k1 s1_track_counts_off s1_table_select s1_table_insert_k1 s1_table_update_k1 s2_table_select s1_track_counts_on s1_ff s2_ff s1_table_stats s1_table_select s1_table_update_k1 s1_ff s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +(1 row) + +step s1_table_insert_k1: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1); +step s1_table_delete_k1: DELETE FROM test_stat_tab WHERE key = 'k1'; +step s1_track_counts_off: SET track_counts = off; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +(1 row) + +step s1_table_insert_k1: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s2_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +k1 | 2 +(2 rows) + +step s1_track_counts_on: SET track_counts = on; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 3| 5| 2| 0| 1| 1| 1| 0 +(1 row) + +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +k1 | 2 +(2 rows) + +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 5| 9| 2| 1| 1| 1| 2| 0 +(1 row) + + +starting permutation: s1_begin s1_table_insert s1_table_update_k1 s1_table_update_k1 s1_table_update_k2 s1_table_update_k2 s1_table_update_k2 s1_table_delete_k1 s1_table_select s1_prepare_a s1_table_select s1_commit_prepared_a s1_table_select s1_ff s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_begin: BEGIN; +step s1_table_insert: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1), ('k2', 1), ('k3', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k2: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k2'; +step s1_table_update_k2: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k2'; +step s1_table_update_k2: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k2'; +step s1_table_delete_k1: DELETE FROM test_stat_tab WHERE key = 'k1'; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +k2 | 4 +k3 | 1 +(3 rows) + +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +(1 row) + +step s1_commit_prepared_a: COMMIT PREPARED 'a'; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +k2 | 4 +k3 | 1 +(3 rows) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 9| 31| 4| 5| 1| 3| 6| 0 +(1 row) + + +starting permutation: s1_begin s1_table_insert s1_table_update_k1 s1_table_update_k1 s1_table_update_k2 s1_table_update_k2 s1_table_update_k2 s1_table_delete_k1 s1_table_select s1_prepare_a s1_table_select s2_commit_prepared_a s1_table_select s1_ff s2_ff s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_begin: BEGIN; +step s1_table_insert: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1), ('k2', 1), ('k3', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k2: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k2'; +step s1_table_update_k2: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k2'; +step s1_table_update_k2: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k2'; +step s1_table_delete_k1: DELETE FROM test_stat_tab WHERE key = 'k1'; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +k2 | 4 +k3 | 1 +(3 rows) + +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +(1 row) + +step s2_commit_prepared_a: COMMIT PREPARED 'a'; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +k2 | 4 +k3 | 1 +(3 rows) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 9| 31| 4| 5| 1| 3| 6| 0 +(1 row) + + +starting permutation: s1_begin s1_table_insert s1_table_update_k1 s1_table_update_k1 s1_table_update_k2 s1_table_update_k2 s1_table_update_k2 s1_table_delete_k1 s1_table_select s1_prepare_a s1_table_select s1_rollback_prepared_a s1_table_select s1_ff s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_begin: BEGIN; +step s1_table_insert: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1), ('k2', 1), ('k3', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k2: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k2'; +step s1_table_update_k2: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k2'; +step s1_table_update_k2: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k2'; +step s1_table_delete_k1: DELETE FROM test_stat_tab WHERE key = 'k1'; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +k2 | 4 +k3 | 1 +(3 rows) + +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +(1 row) + +step s1_rollback_prepared_a: ROLLBACK PREPARED 'a'; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 9| 29| 4| 5| 1| 1| 8| 0 +(1 row) + + +starting permutation: s1_begin s1_table_insert s1_table_update_k1 s1_table_update_k1 s1_table_update_k2 s1_table_update_k2 s1_table_update_k2 s1_table_delete_k1 s1_table_select s1_prepare_a s1_table_select s2_rollback_prepared_a s1_table_select s1_ff s2_ff s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_begin: BEGIN; +step s1_table_insert: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1), ('k2', 1), ('k3', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k2: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k2'; +step s1_table_update_k2: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k2'; +step s1_table_update_k2: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k2'; +step s1_table_delete_k1: DELETE FROM test_stat_tab WHERE key = 'k1'; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +k2 | 4 +k3 | 1 +(3 rows) + +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +(1 row) + +step s2_rollback_prepared_a: ROLLBACK PREPARED 'a'; +step s1_table_select: SELECT * FROM test_stat_tab ORDER BY key, value; +key|value +---+----- +k0 | 1 +(1 row) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 9| 29| 4| 5| 1| 1| 8| 0 +(1 row) + + +starting permutation: s1_table_insert s1_begin s1_table_update_k1 s1_table_update_k1 s1_table_truncate s1_table_insert_k1 s1_table_update_k1 s1_prepare_a s1_commit_prepared_a s1_ff s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_insert: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1), ('k2', 1), ('k3', 1); +step s1_begin: BEGIN; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_truncate: TRUNCATE test_stat_tab; +step s1_table_insert_k1: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s1_commit_prepared_a: COMMIT PREPARED 'a'; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 3| 9| 5| 1| 0| 1| 1| 0 +(1 row) + + +starting permutation: s1_table_insert s1_begin s1_table_update_k1 s1_table_update_k1 s1_table_truncate s1_table_insert_k1 s1_table_update_k1 s1_prepare_a s2_commit_prepared_a s1_ff s2_ff s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_insert: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1), ('k2', 1), ('k3', 1); +step s1_begin: BEGIN; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_truncate: TRUNCATE test_stat_tab; +step s1_table_insert_k1: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s2_commit_prepared_a: COMMIT PREPARED 'a'; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 3| 9| 5| 1| 0| 1| 1| 0 +(1 row) + + +starting permutation: s1_table_insert s1_begin s1_table_update_k1 s1_table_update_k1 s1_table_truncate s1_table_insert_k1 s1_table_update_k1 s1_prepare_a s1_rollback_prepared_a s1_ff s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_insert: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1), ('k2', 1), ('k3', 1); +step s1_begin: BEGIN; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_truncate: TRUNCATE test_stat_tab; +step s1_table_insert_k1: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s1_rollback_prepared_a: ROLLBACK PREPARED 'a'; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 3| 9| 4| 2| 0| 4| 2| 0 +(1 row) + + +starting permutation: s1_table_insert s1_begin s1_table_update_k1 s1_table_update_k1 s1_table_truncate s1_table_insert_k1 s1_table_update_k1 s1_prepare_a s2_rollback_prepared_a s1_ff s2_ff s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_insert: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1), ('k2', 1), ('k3', 1); +step s1_begin: BEGIN; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_truncate: TRUNCATE test_stat_tab; +step s1_table_insert_k1: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s2_rollback_prepared_a: ROLLBACK PREPARED 'a'; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 3| 9| 4| 2| 0| 4| 2| 0 +(1 row) + + +starting permutation: s1_table_insert s1_table_update_k1 s1_begin s1_table_delete_k1 s1_table_insert_k1 s1_table_update_k1 s1_table_update_k1 s1_table_drop s1_prepare_a s1_rollback_prepared_a s1_ff s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_insert: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1), ('k2', 1), ('k3', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_begin: BEGIN; +step s1_table_delete_k1: DELETE FROM test_stat_tab WHERE key = 'k1'; +step s1_table_insert_k1: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_drop: DROP TABLE test_stat_tab; +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s1_rollback_prepared_a: ROLLBACK PREPARED 'a'; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 4| 16| 5| 3| 1| 4| 4| 0 +(1 row) + + +starting permutation: s1_table_insert s1_table_update_k1 s1_begin s1_table_delete_k1 s1_table_insert_k1 s1_table_update_k1 s1_table_update_k1 s1_table_drop s1_prepare_a s2_rollback_prepared_a s1_ff s2_ff s1_table_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_insert: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1), ('k2', 1), ('k3', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_begin: BEGIN; +step s1_table_delete_k1: DELETE FROM test_stat_tab WHERE key = 'k1'; +step s1_table_insert_k1: INSERT INTO test_stat_tab(key, value) VALUES('k1', 1); +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_update_k1: UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1'; +step s1_table_drop: DROP TABLE test_stat_tab; +step s1_prepare_a: PREPARE TRANSACTION 'a'; +step s2_rollback_prepared_a: ROLLBACK PREPARED 'a'; +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_table_stats: + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' + +seq_scan|seq_tup_read|n_tup_ins|n_tup_upd|n_tup_del|n_live_tup|n_dead_tup|vacuum_count +--------+------------+---------+---------+---------+----------+----------+------------ + 4| 16| 5| 3| 1| 4| 4| 0 +(1 row) + + +starting permutation: s1_slru_save_stats s1_listen s1_begin s1_big_notify s1_ff s1_slru_check_stats s1_commit s1_slru_check_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_save_stats: + INSERT INTO test_slru_stats VALUES('Notify', 'blks_zeroed', + (SELECT blks_zeroed FROM pg_stat_slru WHERE name = 'Notify')); + +step s1_listen: LISTEN stats_test_nothing; +step s1_begin: BEGIN; +step s1_big_notify: SELECT pg_notify('stats_test_use', + repeat('0', current_setting('block_size')::int / 2)) FROM generate_series(1, 2); + +pg_notify +--------- + + +(2 rows) + +step s1_ff: SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +f +(1 row) + +step s1_commit: COMMIT; +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +t +(1 row) + + +starting permutation: s1_slru_save_stats s1_listen s2_big_notify s2_ff s1_slru_check_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_save_stats: + INSERT INTO test_slru_stats VALUES('Notify', 'blks_zeroed', + (SELECT blks_zeroed FROM pg_stat_slru WHERE name = 'Notify')); + +step s1_listen: LISTEN stats_test_nothing; +step s2_big_notify: SELECT pg_notify('stats_test_use', + repeat('0', current_setting('block_size')::int / 2)) FROM generate_series(1, 2); + +pg_notify +--------- + + +(2 rows) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +t +(1 row) + + +starting permutation: s1_slru_save_stats s1_listen s2_begin s2_big_notify s2_ff s1_slru_check_stats s2_commit +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_save_stats: + INSERT INTO test_slru_stats VALUES('Notify', 'blks_zeroed', + (SELECT blks_zeroed FROM pg_stat_slru WHERE name = 'Notify')); + +step s1_listen: LISTEN stats_test_nothing; +step s2_begin: BEGIN; +step s2_big_notify: SELECT pg_notify('stats_test_use', + repeat('0', current_setting('block_size')::int / 2)) FROM generate_series(1, 2); + +pg_notify +--------- + + +(2 rows) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +f +(1 row) + +step s2_commit: COMMIT; + +starting permutation: s1_fetch_consistency_none s1_slru_save_stats s1_listen s1_begin s1_slru_check_stats s2_big_notify s2_ff s1_slru_check_stats s1_commit s1_slru_check_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_fetch_consistency_none: SET stats_fetch_consistency = 'none'; +step s1_slru_save_stats: + INSERT INTO test_slru_stats VALUES('Notify', 'blks_zeroed', + (SELECT blks_zeroed FROM pg_stat_slru WHERE name = 'Notify')); + +step s1_listen: LISTEN stats_test_nothing; +step s1_begin: BEGIN; +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +f +(1 row) + +step s2_big_notify: SELECT pg_notify('stats_test_use', + repeat('0', current_setting('block_size')::int / 2)) FROM generate_series(1, 2); + +pg_notify +--------- + + +(2 rows) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +t +(1 row) + +step s1_commit: COMMIT; +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +t +(1 row) + + +starting permutation: s1_fetch_consistency_cache s1_slru_save_stats s1_listen s1_begin s1_slru_check_stats s2_big_notify s2_ff s1_slru_check_stats s1_commit s1_slru_check_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_fetch_consistency_cache: SET stats_fetch_consistency = 'cache'; +step s1_slru_save_stats: + INSERT INTO test_slru_stats VALUES('Notify', 'blks_zeroed', + (SELECT blks_zeroed FROM pg_stat_slru WHERE name = 'Notify')); + +step s1_listen: LISTEN stats_test_nothing; +step s1_begin: BEGIN; +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +f +(1 row) + +step s2_big_notify: SELECT pg_notify('stats_test_use', + repeat('0', current_setting('block_size')::int / 2)) FROM generate_series(1, 2); + +pg_notify +--------- + + +(2 rows) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +f +(1 row) + +step s1_commit: COMMIT; +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +t +(1 row) + + +starting permutation: s1_fetch_consistency_snapshot s1_slru_save_stats s1_listen s1_begin s1_slru_check_stats s2_big_notify s2_ff s1_slru_check_stats s1_commit s1_slru_check_stats +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_fetch_consistency_snapshot: SET stats_fetch_consistency = 'snapshot'; +step s1_slru_save_stats: + INSERT INTO test_slru_stats VALUES('Notify', 'blks_zeroed', + (SELECT blks_zeroed FROM pg_stat_slru WHERE name = 'Notify')); + +step s1_listen: LISTEN stats_test_nothing; +step s1_begin: BEGIN; +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +f +(1 row) + +step s2_big_notify: SELECT pg_notify('stats_test_use', + repeat('0', current_setting('block_size')::int / 2)) FROM generate_series(1, 2); + +pg_notify +--------- + + +(2 rows) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +f +(1 row) + +step s1_commit: COMMIT; +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +t +(1 row) + + +starting permutation: s1_fetch_consistency_none s1_slru_save_stats s1_listen s1_begin s1_slru_check_stats s2_big_notify s2_ff s1_slru_check_stats s1_clear_snapshot s1_slru_check_stats s1_commit +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_fetch_consistency_none: SET stats_fetch_consistency = 'none'; +step s1_slru_save_stats: + INSERT INTO test_slru_stats VALUES('Notify', 'blks_zeroed', + (SELECT blks_zeroed FROM pg_stat_slru WHERE name = 'Notify')); + +step s1_listen: LISTEN stats_test_nothing; +step s1_begin: BEGIN; +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +f +(1 row) + +step s2_big_notify: SELECT pg_notify('stats_test_use', + repeat('0', current_setting('block_size')::int / 2)) FROM generate_series(1, 2); + +pg_notify +--------- + + +(2 rows) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +t +(1 row) + +step s1_clear_snapshot: SELECT pg_stat_clear_snapshot(); +pg_stat_clear_snapshot +---------------------- + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +t +(1 row) + +step s1_commit: COMMIT; + +starting permutation: s1_fetch_consistency_cache s1_slru_save_stats s1_listen s1_begin s1_slru_check_stats s2_big_notify s2_ff s1_slru_check_stats s1_clear_snapshot s1_slru_check_stats s1_commit +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_fetch_consistency_cache: SET stats_fetch_consistency = 'cache'; +step s1_slru_save_stats: + INSERT INTO test_slru_stats VALUES('Notify', 'blks_zeroed', + (SELECT blks_zeroed FROM pg_stat_slru WHERE name = 'Notify')); + +step s1_listen: LISTEN stats_test_nothing; +step s1_begin: BEGIN; +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +f +(1 row) + +step s2_big_notify: SELECT pg_notify('stats_test_use', + repeat('0', current_setting('block_size')::int / 2)) FROM generate_series(1, 2); + +pg_notify +--------- + + +(2 rows) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +f +(1 row) + +step s1_clear_snapshot: SELECT pg_stat_clear_snapshot(); +pg_stat_clear_snapshot +---------------------- + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +t +(1 row) + +step s1_commit: COMMIT; + +starting permutation: s1_fetch_consistency_snapshot s1_slru_save_stats s1_listen s1_begin s1_slru_check_stats s2_big_notify s2_ff s1_slru_check_stats s1_clear_snapshot s1_slru_check_stats s1_commit +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_fetch_consistency_snapshot: SET stats_fetch_consistency = 'snapshot'; +step s1_slru_save_stats: + INSERT INTO test_slru_stats VALUES('Notify', 'blks_zeroed', + (SELECT blks_zeroed FROM pg_stat_slru WHERE name = 'Notify')); + +step s1_listen: LISTEN stats_test_nothing; +step s1_begin: BEGIN; +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +f +(1 row) + +step s2_big_notify: SELECT pg_notify('stats_test_use', + repeat('0', current_setting('block_size')::int / 2)) FROM generate_series(1, 2); + +pg_notify +--------- + + +(2 rows) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +f +(1 row) + +step s1_clear_snapshot: SELECT pg_stat_clear_snapshot(); +pg_stat_clear_snapshot +---------------------- + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +t +(1 row) + +step s1_commit: COMMIT; + +starting permutation: s1_fetch_consistency_snapshot s1_slru_save_stats s1_listen s1_begin s1_func_stats s2_big_notify s2_ff s1_slru_check_stats s1_commit +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_fetch_consistency_snapshot: SET stats_fetch_consistency = 'snapshot'; +step s1_slru_save_stats: + INSERT INTO test_slru_stats VALUES('Notify', 'blks_zeroed', + (SELECT blks_zeroed FROM pg_stat_slru WHERE name = 'Notify')); + +step s1_listen: LISTEN stats_test_nothing; +step s1_begin: BEGIN; +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s2_big_notify: SELECT pg_notify('stats_test_use', + repeat('0', current_setting('block_size')::int / 2)) FROM generate_series(1, 2); + +pg_notify +--------- + + +(2 rows) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +f +(1 row) + +step s1_commit: COMMIT; + +starting permutation: s1_fetch_consistency_snapshot s1_slru_save_stats s1_listen s1_begin s2_big_notify s2_ff s1_slru_check_stats s2_func_call s2_ff s1_func_stats s1_clear_snapshot s1_func_stats s1_commit +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_fetch_consistency_snapshot: SET stats_fetch_consistency = 'snapshot'; +step s1_slru_save_stats: + INSERT INTO test_slru_stats VALUES('Notify', 'blks_zeroed', + (SELECT blks_zeroed FROM pg_stat_slru WHERE name = 'Notify')); + +step s1_listen: LISTEN stats_test_nothing; +step s1_begin: BEGIN; +step s2_big_notify: SELECT pg_notify('stats_test_use', + repeat('0', current_setting('block_size')::int / 2)) FROM generate_series(1, 2); + +pg_notify +--------- + + +(2 rows) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_slru_check_stats: + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + +?column? +-------- +t +(1 row) + +step s2_func_call: SELECT test_stat_func() +test_stat_func +-------------- + +(1 row) + +step s2_ff: SELECT pg_stat_force_next_flush(); +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| | | +(1 row) + +step s1_clear_snapshot: SELECT pg_stat_clear_snapshot(); +pg_stat_clear_snapshot +---------------------- + +(1 row) + +step s1_func_stats: + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' + +name |pg_stat_get_function_calls|total_above_zero|self_above_zero +--------------+--------------------------+----------------+--------------- +test_stat_func| 1|t |t +(1 row) + +step s1_commit: COMMIT; diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index a48caae228e..c3066a6748d 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -89,6 +89,7 @@ test: timeouts test: vacuum-concurrent-drop test: vacuum-conflict test: vacuum-skip-locked +test: stats test: horizons test: predicate-hash test: predicate-gist diff --git a/src/test/isolation/specs/stats.spec b/src/test/isolation/specs/stats.spec new file mode 100644 index 00000000000..cc57d461f65 --- /dev/null +++ b/src/test/isolation/specs/stats.spec @@ -0,0 +1,753 @@ +setup +{ + CREATE TABLE test_stat_oid(name text NOT NULL, oid oid); + + CREATE TABLE test_stat_tab(key text not null, value int); + INSERT INTO test_stat_tab(key, value) VALUES('k0', 1); + INSERT INTO test_stat_oid(name, oid) VALUES('test_stat_tab', 'test_stat_tab'::regclass); + + CREATE FUNCTION test_stat_func() RETURNS VOID LANGUAGE plpgsql AS $$BEGIN END;$$; + INSERT INTO test_stat_oid(name, oid) VALUES('test_stat_func', 'test_stat_func'::regproc); + + CREATE FUNCTION test_stat_func2() RETURNS VOID LANGUAGE plpgsql AS $$BEGIN END;$$; + INSERT INTO test_stat_oid(name, oid) VALUES('test_stat_func2', 'test_stat_func2'::regproc); + + CREATE TABLE test_slru_stats(slru TEXT, stat TEXT, value INT); + + SELECT pg_stat_force_next_flush(); +} + +teardown +{ + DROP TABLE test_stat_oid; + DROP TABLE test_slru_stats; + + DROP TABLE IF EXISTS test_stat_tab; + DROP FUNCTION IF EXISTS test_stat_func(); + DROP FUNCTION IF EXISTS test_stat_func2(); +} + +session s1 +setup { SET stats_fetch_consistency = 'none'; } +step s1_fetch_consistency_none { SET stats_fetch_consistency = 'none'; } +step s1_fetch_consistency_cache { SET stats_fetch_consistency = 'cache'; } +step s1_fetch_consistency_snapshot { SET stats_fetch_consistency = 'snapshot'; } +step s1_disable_debug_discard { SET debug_discard_caches = 0; } +step s1_clear_snapshot { SELECT pg_stat_clear_snapshot(); } +step s1_begin { BEGIN; } +step s1_commit { COMMIT; } +step s1_rollback { ROLLBACK; } +step s1_prepare_a { PREPARE TRANSACTION 'a'; } +step s1_commit_prepared_a { COMMIT PREPARED 'a'; } +step s1_rollback_prepared_a { ROLLBACK PREPARED 'a'; } + +# Function stats steps +step s1_ff { SELECT pg_stat_force_next_flush(); RESET debug_discard_caches; } +step s1_track_funcs_all { SET track_functions = 'all'; } +step s1_track_funcs_none { SET track_functions = 'none'; } +step s1_func_call { SELECT test_stat_func(); } +step s1_func_drop { DROP FUNCTION test_stat_func(); } +step s1_func_stats_reset { SELECT pg_stat_reset_single_function_counters('test_stat_func'::regproc); } +step s1_func_stats_reset_nonexistent { SELECT pg_stat_reset_single_function_counters(12000); } +step s1_reset { SELECT pg_stat_reset(); } +step s1_func_stats { + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' +} +step s1_func_stats2 { + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func2' +} +step s1_func_stats_nonexistent { + SELECT pg_stat_get_function_calls(12000); +} + +# Relation stats steps +step s1_track_counts_on { SET track_counts = on; } +step s1_track_counts_off { SET track_counts = off; } +step s1_table_select { SELECT * FROM test_stat_tab ORDER BY key, value; } +step s1_table_insert { INSERT INTO test_stat_tab(key, value) VALUES('k1', 1), ('k2', 1), ('k3', 1);} +step s1_table_insert_k1 { INSERT INTO test_stat_tab(key, value) VALUES('k1', 1);} +step s1_table_update_k1 { UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1';} +step s1_table_update_k2 { UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k2';} +step s1_table_delete_k1 { DELETE FROM test_stat_tab WHERE key = 'k1';} +step s1_table_truncate { TRUNCATE test_stat_tab; } +step s1_table_drop { DROP TABLE test_stat_tab; } + +step s1_table_stats { + SELECT + pg_stat_get_numscans(tso.oid) AS seq_scan, + pg_stat_get_tuples_returned(tso.oid) AS seq_tup_read, + pg_stat_get_tuples_inserted(tso.oid) AS n_tup_ins, + pg_stat_get_tuples_updated(tso.oid) AS n_tup_upd, + pg_stat_get_tuples_deleted(tso.oid) AS n_tup_del, + pg_stat_get_live_tuples(tso.oid) AS n_live_tup, + pg_stat_get_dead_tuples(tso.oid) AS n_dead_tup, + pg_stat_get_vacuum_count(tso.oid) AS vacuum_count + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_tab' +} + +# SLRU stats steps +step s1_slru_save_stats { + INSERT INTO test_slru_stats VALUES('Notify', 'blks_zeroed', + (SELECT blks_zeroed FROM pg_stat_slru WHERE name = 'Notify')); +} +step s1_listen { LISTEN stats_test_nothing; } +step s1_big_notify { SELECT pg_notify('stats_test_use', + repeat('0', current_setting('block_size')::int / 2)) FROM generate_series(1, 2); + } + +step s1_slru_check_stats { + SELECT current.blks_zeroed > before.value + FROM test_slru_stats before + INNER JOIN pg_stat_slru current + ON before.slru = current.name + WHERE before.stat = 'blks_zeroed'; + } + + +session s2 +setup { SET stats_fetch_consistency = 'none'; RESET debug_discard_caches; } +step s2_begin { BEGIN; } +step s2_disable_debug_discard { SET debug_discard_caches = 0; } +step s2_commit { COMMIT; } +step s2_commit_prepared_a { COMMIT PREPARED 'a'; } +step s2_rollback_prepared_a { ROLLBACK PREPARED 'a'; } +step s2_ff { SELECT pg_stat_force_next_flush(); } + +# Function stats steps +step s2_track_funcs_all { SET track_functions = 'all'; } +step s2_track_funcs_none { SET track_functions = 'none'; } +step s2_func_call { SELECT test_stat_func() } +step s2_func_call2 { SELECT test_stat_func2() } +step s2_func_stats { + SELECT + tso.name, + pg_stat_get_function_calls(tso.oid), + pg_stat_get_function_total_time(tso.oid) > 0 total_above_zero, + pg_stat_get_function_self_time(tso.oid) > 0 self_above_zero + FROM test_stat_oid AS tso + WHERE tso.name = 'test_stat_func' +} + +# Relation stats steps +step s2_table_select { SELECT * FROM test_stat_tab ORDER BY key, value; } +step s2_table_update_k1 { UPDATE test_stat_tab SET value = value + 1 WHERE key = 'k1';} + +# SLRU stats steps +step s2_big_notify { SELECT pg_notify('stats_test_use', + repeat('0', current_setting('block_size')::int / 2)) FROM generate_series(1, 2); + } + + +###################### +# Function stats tests +###################### + +# check that stats are collected iff enabled +permutation + s1_track_funcs_none s1_func_stats s1_func_call s1_func_call s1_ff s1_func_stats +permutation + s1_track_funcs_all s1_func_stats s1_func_call s1_func_call s1_ff s1_func_stats + +# multiple function calls are accurately reported, across separate connections +permutation + s1_track_funcs_all s2_track_funcs_all s1_func_stats s2_func_stats + s1_func_call s2_func_call s1_func_call s2_func_call s2_func_call s1_ff s2_ff s1_func_stats s2_func_stats +permutation + s1_track_funcs_all s2_track_funcs_all s1_func_stats s2_func_stats + s1_func_call s1_ff s2_func_call s2_func_call s2_ff s1_func_stats s2_func_stats +permutation + s1_track_funcs_all s2_track_funcs_all s1_func_stats s2_func_stats + s1_begin s1_func_call s1_func_call s1_commit s1_ff s1_func_stats s2_func_stats + + +### Check interaction between dropping and stats reporting + +# Disable debug_discard_caches for a few of these tests - we precisely are +# testing the behaviour of no invalidations arriving. "Real" invalidations +# shouldn't trigger behavioural difference, because we are testing paths +# precisely because they do not have AcceptInvalidationMessages calls.() + +# dropping a table remove stats iff committed +permutation + s1_track_funcs_all s2_track_funcs_all s1_func_stats s2_func_stats + s1_begin s1_func_call s2_func_call s1_func_drop s2_func_call s2_ff s2_func_stats s1_commit s1_ff s1_func_stats s2_func_stats +permutation + s1_track_funcs_all s2_track_funcs_all s1_func_stats s2_func_stats + s1_begin s1_func_call s2_func_call s1_func_drop s2_func_call s2_ff s2_func_stats s1_rollback s1_ff s1_func_stats s2_func_stats + +# Verify that pending stats from before a drop do not lead to +# reviving stats for a dropped object +permutation + s1_track_funcs_all s2_track_funcs_all + s2_func_call s2_ff # this access increments refcount, preventing the shared entry from being dropped + s2_begin s2_func_call s1_func_drop s1_func_stats s2_commit s2_ff s1_func_stats s2_func_stats +permutation + s1_track_funcs_all s2_track_funcs_all + s2_begin s2_func_call s1_func_drop s1_func_stats s2_commit s2_ff s1_func_stats s2_func_stats +permutation + s1_disable_debug_discard s2_disable_debug_discard + s1_track_funcs_all s2_track_funcs_all + s1_func_call s2_begin s2_func_call s1_func_drop s2_func_call s2_commit s2_ff s1_func_stats s2_func_stats + +# Function calls don't necessarily trigger cache invalidation processing. The +# default handling of dropped stats could therefore end up with stats getting +# revived by a function call done after stats processing - but +# pgstat_init_function_usage() protects against that if track_functions is +# on. Verify that the stats are indeed dropped, and document the behavioural +# difference between track_functions settings. +permutation + s1_disable_debug_discard s2_disable_debug_discard + s1_track_funcs_all s2_track_funcs_none + s1_func_call s2_begin s2_func_call s1_ff s1_func_stats s1_func_drop s2_track_funcs_none s1_func_stats s2_func_call s2_commit s2_ff s1_func_stats s2_func_stats +permutation + s1_disable_debug_discard s2_disable_debug_discard + s1_track_funcs_all s2_track_funcs_none + s1_func_call s2_begin s2_func_call s1_ff s1_func_stats s1_func_drop s2_track_funcs_all s1_func_stats s2_func_call s2_commit s2_ff s1_func_stats s2_func_stats + +# test pg_stat_reset_single_function_counters +permutation + s1_track_funcs_all s2_track_funcs_all + s1_func_call + s2_func_call + s2_func_call2 + s1_ff s2_ff + s1_func_stats + s2_func_call s2_func_call2 s2_ff + s1_func_stats s1_func_stats2 s1_func_stats + s1_func_stats_reset + s1_func_stats s1_func_stats2 s1_func_stats + +# test pg_stat_reset_single_function_counters of non-existing function +permutation + s1_func_stats_nonexistent + s1_func_stats_reset_nonexistent + s1_func_stats_nonexistent + +# test pg_stat_reset +permutation + s1_track_funcs_all s2_track_funcs_all + s1_func_call + s2_func_call + s2_func_call2 + s1_ff s2_ff + s1_func_stats s1_func_stats2 s1_func_stats + s1_reset + s1_func_stats s1_func_stats2 s1_func_stats + + +### Check the different snapshot consistency models + +# First just some dead-trivial test verifying each model doesn't crash +permutation + s1_track_funcs_all s1_fetch_consistency_none s1_func_call s1_ff s1_func_stats +permutation + s1_track_funcs_all s1_fetch_consistency_cache s1_func_call s1_ff s1_func_stats +permutation + s1_track_funcs_all s1_fetch_consistency_snapshot s1_func_call s1_ff s1_func_stats + +# with stats_fetch_consistency=none s1 should see flushed changes in s2, despite being in a transaction +permutation + s1_track_funcs_all s2_track_funcs_all + s1_fetch_consistency_none + s2_func_call s2_ff + s1_begin + s1_func_stats + s2_func_call s2_ff + s1_func_stats + s1_commit + +# with stats_fetch_consistency=cache s1 should not see concurrent +# changes to the same object after the first access, but a separate +# object should show changes +permutation + s1_track_funcs_all s2_track_funcs_all + s1_fetch_consistency_cache + s2_func_call s2_func_call2 s2_ff + s1_begin + s1_func_stats + s2_func_call s2_func_call2 s2_ff + s1_func_stats s1_func_stats2 + s1_commit + +# with stats_fetch_consistency=snapshot s1 should not see any +# concurrent changes after the first access +permutation + s1_track_funcs_all s2_track_funcs_all + s1_fetch_consistency_snapshot + s2_func_call s2_func_call2 s2_ff + s1_begin + s1_func_stats + s2_func_call s2_func_call2 s2_ff + s1_func_stats s1_func_stats2 + s1_commit + +# Check access to non-existing stats works correctly and repeatedly +permutation + s1_fetch_consistency_none + s1_begin + s1_func_stats_nonexistent + s1_func_stats_nonexistent + s1_commit +permutation + s1_fetch_consistency_cache + s1_begin + s1_func_stats_nonexistent + s1_func_stats_nonexistent + s1_commit +permutation + s1_fetch_consistency_snapshot + s1_begin + s1_func_stats_nonexistent + s1_func_stats_nonexistent + s1_commit + + +### Check 2PC handling of stat drops + +# S1 prepared, S1 commits prepared +permutation + s1_track_funcs_all s2_track_funcs_all + s1_begin + s1_func_call + s2_func_call + s1_func_drop + s2_func_call + s2_ff + s1_prepare_a + s2_func_call + s2_ff + s1_func_call + s1_ff + s1_func_stats + s1_commit_prepared_a + s1_func_stats + +# S1 prepared, S1 aborts prepared +permutation + s1_track_funcs_all s2_track_funcs_all + s1_begin + s1_func_call + s2_func_call + s1_func_drop + s2_func_call + s2_ff + s1_prepare_a + s2_func_call + s2_ff + s1_func_call + s1_ff + s1_func_stats + s1_rollback_prepared_a + s1_func_stats + +# S1 prepares, S2 commits prepared +permutation + s1_track_funcs_all s2_track_funcs_all + s1_begin + s1_func_call + s2_func_call + s1_func_drop + s2_func_call + s2_ff + s1_prepare_a + s2_func_call + s2_ff + s1_func_call + s1_ff + s1_func_stats + s2_commit_prepared_a + s1_func_stats + +# S1 prepared, S2 aborts prepared +permutation + s1_track_funcs_all s2_track_funcs_all + s1_begin + s1_func_call + s2_func_call + s1_func_drop + s2_func_call + s2_ff + s1_prepare_a + s2_func_call + s2_ff + s1_func_call + s1_ff + s1_func_stats + s2_rollback_prepared_a + s1_func_stats + + +###################### +# Table stats tests +###################### + +# Most of the stats handling mechanism has already been tested in the function +# stats tests above - that's cheaper than testing with relations. But +# particularly for 2PC there are special cases + + +### Verify that pending stats from before a drop do not lead to reviving +### of stats for a dropped object + +permutation + s1_table_select + s1_table_insert + s2_table_select + s2_table_update_k1 + s1_ff + s2_table_update_k1 + s1_table_drop + s2_ff + s1_table_stats + +permutation + s1_table_select + s1_table_insert + s2_table_select + s2_table_update_k1 + s2_table_update_k1 + s1_table_drop + s1_table_stats + + +### Check that we don't count changes with track counts off, but allow access +### to prior stats + +# simple read access with stats off +permutation + s1_track_counts_off + s1_table_stats + s1_track_counts_on + +# simple read access with stats off, previously accessed +permutation + s1_table_select + s1_track_counts_off + s1_ff + s1_table_stats + s1_track_counts_on +permutation + s1_table_select + s1_ff + s1_track_counts_off + s1_table_stats + s1_track_counts_on + +# ensure we don't count anything with stats off +permutation + s1_track_counts_off + s1_table_select + s1_table_insert_k1 + s1_table_update_k1 + s2_table_select + s1_track_counts_on + s1_ff s2_ff + s1_table_stats + # but can count again after + s1_table_select + s1_table_update_k1 + s1_ff + s1_table_stats +permutation + s1_table_select + s1_table_insert_k1 + s1_table_delete_k1 + s1_track_counts_off + s1_table_select + s1_table_insert_k1 + s1_table_update_k1 + s2_table_select + s1_track_counts_on + s1_ff s2_ff + s1_table_stats + s1_table_select + s1_table_update_k1 + s1_ff + s1_table_stats + + +### 2PC: transactional and non-transactional counters work correctly + +# S1 prepares, S2 commits prepared +permutation + s1_begin + s1_table_insert s1_table_update_k1 s1_table_update_k1 s1_table_update_k2 s1_table_update_k2 s1_table_update_k2 s1_table_delete_k1 + s1_table_select + s1_prepare_a + s1_table_select + s1_commit_prepared_a + s1_table_select + s1_ff + s1_table_stats + +# S1 prepares, S2 commits prepared +permutation + s1_begin + s1_table_insert s1_table_update_k1 s1_table_update_k1 s1_table_update_k2 s1_table_update_k2 s1_table_update_k2 s1_table_delete_k1 + s1_table_select + s1_prepare_a + s1_table_select + s2_commit_prepared_a + s1_table_select + s1_ff s2_ff + s1_table_stats + +# S1 prepares, S2 commits prepared +permutation + s1_begin + s1_table_insert s1_table_update_k1 s1_table_update_k1 s1_table_update_k2 s1_table_update_k2 s1_table_update_k2 s1_table_delete_k1 + s1_table_select + s1_prepare_a + s1_table_select + s1_rollback_prepared_a + s1_table_select + s1_ff + s1_table_stats + +# S1 prepares, S1 aborts prepared +permutation + s1_begin + s1_table_insert s1_table_update_k1 s1_table_update_k1 s1_table_update_k2 s1_table_update_k2 s1_table_update_k2 s1_table_delete_k1 + s1_table_select + s1_prepare_a + s1_table_select + s2_rollback_prepared_a + s1_table_select + s1_ff s2_ff + s1_table_stats + + +### 2PC: truncate handling + +# S1 prepares, S1 commits prepared +permutation + s1_table_insert + s1_begin + s1_table_update_k1 # should *not* be counted, different rel + s1_table_update_k1 # dito + s1_table_truncate + s1_table_insert_k1 # should be counted + s1_table_update_k1 # dito + s1_prepare_a + s1_commit_prepared_a + s1_ff + s1_table_stats + +# S1 prepares, S2 commits prepared +permutation + s1_table_insert + s1_begin + s1_table_update_k1 # should *not* be counted, different rel + s1_table_update_k1 # dito + s1_table_truncate + s1_table_insert_k1 # should be counted + s1_table_update_k1 # dito + s1_prepare_a + s2_commit_prepared_a + s1_ff s2_ff + s1_table_stats + +# S1 prepares, S1 aborts prepared +permutation + s1_table_insert + s1_begin + s1_table_update_k1 # should be counted + s1_table_update_k1 # dito + s1_table_truncate + s1_table_insert_k1 # should *not* be counted, different rel + s1_table_update_k1 # dito + s1_prepare_a + s1_rollback_prepared_a + s1_ff + s1_table_stats + +# S1 prepares, S2 aborts prepared +permutation + s1_table_insert + s1_begin + s1_table_update_k1 # should be counted + s1_table_update_k1 # dito + s1_table_truncate + s1_table_insert_k1 # should *not* be counted, different rel + s1_table_update_k1 # dito + s1_prepare_a + s2_rollback_prepared_a + s1_ff s2_ff + s1_table_stats + + +### 2PC: rolled back drop maintains live / dead counters + +# S1 prepares, S1 aborts prepared +permutation + s1_table_insert + s1_table_update_k1 + s1_begin + # should all be counted + s1_table_delete_k1 + s1_table_insert_k1 + s1_table_update_k1 + s1_table_update_k1 + s1_table_drop + s1_prepare_a + s1_rollback_prepared_a + s1_ff + s1_table_stats + +# S1 prepares, S1 aborts prepared +permutation + s1_table_insert + s1_table_update_k1 + s1_begin + # should all be counted + s1_table_delete_k1 + s1_table_insert_k1 + s1_table_update_k1 + s1_table_update_k1 + s1_table_drop + s1_prepare_a + s2_rollback_prepared_a + s1_ff s2_ff + s1_table_stats + + +###################### +# SLRU stats tests +###################### + +# Verify SLRU stats generated in own transaction +permutation + s1_slru_save_stats + s1_listen + s1_begin + s1_big_notify + s1_ff + s1_slru_check_stats + s1_commit + s1_slru_check_stats + +# Verify SLRU stats generated in separate transaction +permutation + s1_slru_save_stats + s1_listen + s2_big_notify + s2_ff + s1_slru_check_stats + +# shouldn't see stats yet, not committed +permutation + s1_slru_save_stats + s1_listen + s2_begin + s2_big_notify + s2_ff + s1_slru_check_stats + s2_commit + + +### Check the different snapshot consistency models for fixed-amount statistics + +permutation + s1_fetch_consistency_none + s1_slru_save_stats s1_listen + s1_begin + s1_slru_check_stats + s2_big_notify + s2_ff + s1_slru_check_stats + s1_commit + s1_slru_check_stats +permutation + s1_fetch_consistency_cache + s1_slru_save_stats s1_listen + s1_begin + s1_slru_check_stats + s2_big_notify + s2_ff + s1_slru_check_stats + s1_commit + s1_slru_check_stats +permutation + s1_fetch_consistency_snapshot + s1_slru_save_stats s1_listen + s1_begin + s1_slru_check_stats + s2_big_notify + s2_ff + s1_slru_check_stats + s1_commit + s1_slru_check_stats + +# check that pg_stat_clear_snapshot(), well ... +permutation + s1_fetch_consistency_none + s1_slru_save_stats s1_listen + s1_begin + s1_slru_check_stats + s2_big_notify + s2_ff + s1_slru_check_stats + s1_clear_snapshot + s1_slru_check_stats + s1_commit +permutation + s1_fetch_consistency_cache + s1_slru_save_stats s1_listen + s1_begin + s1_slru_check_stats + s2_big_notify + s2_ff + s1_slru_check_stats + s1_clear_snapshot + s1_slru_check_stats + s1_commit +permutation + s1_fetch_consistency_snapshot + s1_slru_save_stats s1_listen + s1_begin + s1_slru_check_stats + s2_big_notify + s2_ff + s1_slru_check_stats + s1_clear_snapshot + s1_slru_check_stats + s1_commit + +# check that a variable-amount stats access caches fixed-amount stat too +permutation + s1_fetch_consistency_snapshot + s1_slru_save_stats s1_listen + s1_begin + s1_func_stats + s2_big_notify + s2_ff + s1_slru_check_stats + s1_commit + +# and the other way round +permutation + s1_fetch_consistency_snapshot + s1_slru_save_stats s1_listen + s1_begin + s2_big_notify + s2_ff + s1_slru_check_stats + s2_func_call + s2_ff + s1_func_stats + s1_clear_snapshot + s1_func_stats + s1_commit diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out index 1a9b6d9ec9b..890b5d7e5bc 100644 --- a/src/test/regress/expected/stats.out +++ b/src/test/regress/expected/stats.out @@ -16,6 +16,8 @@ SET enable_seqscan TO on; SET enable_indexscan TO on; -- for the moment, we don't want index-only scans here SET enable_indexonlyscan TO off; +-- not enabled by default, but we want to test it... +SET track_functions TO 'all'; -- save counters BEGIN; SET LOCAL stats_fetch_consistency = snapshot; @@ -96,6 +98,108 @@ SELECT count(*) FROM tenk2 WHERE unique1 = 1; (1 row) RESET enable_bitmapscan; +-- Check that stats for tables are dropped. For that we need to access stats +-- by oid after the DROP TABLE. Save oids. +CREATE TABLE drop_stats_test(); +INSERT INTO drop_stats_test DEFAULT VALUES; +SELECT 'drop_stats_test'::regclass::oid AS drop_stats_test_oid \gset +CREATE TABLE drop_stats_test_xact(); +INSERT INTO drop_stats_test_xact DEFAULT VALUES; +SELECT 'drop_stats_test_xact'::regclass::oid AS drop_stats_test_xact_oid \gset +CREATE TABLE drop_stats_test_subxact(); +INSERT INTO drop_stats_test_subxact DEFAULT VALUES; +SELECT 'drop_stats_test_subxact'::regclass::oid AS drop_stats_test_subxact_oid \gset +-- Basic testing for track_functions +CREATE FUNCTION stats_test_func1() RETURNS VOID LANGUAGE plpgsql AS $$BEGIN END;$$; +SELECT 'stats_test_func1()'::regprocedure::oid AS stats_test_func1_oid \gset +CREATE FUNCTION stats_test_func2() RETURNS VOID LANGUAGE plpgsql AS $$BEGIN END;$$; +SELECT 'stats_test_func2()'::regprocedure::oid AS stats_test_func2_oid \gset +-- Basic test that stats are accumulated +BEGIN; +SET LOCAL stats_fetch_consistency = none; +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); + pg_stat_get_function_calls +---------------------------- + +(1 row) + +SELECT pg_stat_get_xact_function_calls(:stats_test_func1_oid); + pg_stat_get_xact_function_calls +--------------------------------- + +(1 row) + +SELECT stats_test_func1(); + stats_test_func1 +------------------ + +(1 row) + +SELECT pg_stat_get_xact_function_calls(:stats_test_func1_oid); + pg_stat_get_xact_function_calls +--------------------------------- + 1 +(1 row) + +SELECT stats_test_func1(); + stats_test_func1 +------------------ + +(1 row) + +SELECT pg_stat_get_xact_function_calls(:stats_test_func1_oid); + pg_stat_get_xact_function_calls +--------------------------------- + 2 +(1 row) + +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); + pg_stat_get_function_calls +---------------------------- + 0 +(1 row) + +COMMIT; +-- Verify that function stats are not transactional (displayed after +-- pg_stat_force_next_flush() below) +-- rolled back savepoint in committing transaction +BEGIN; +SELECT stats_test_func2(); + stats_test_func2 +------------------ + +(1 row) + +SAVEPOINT foo; +SELECT stats_test_func2(); + stats_test_func2 +------------------ + +(1 row) + +ROLLBACK TO SAVEPOINT foo; +SELECT pg_stat_get_xact_function_calls(:stats_test_func2_oid); + pg_stat_get_xact_function_calls +--------------------------------- + 2 +(1 row) + +SELECT stats_test_func2(); + stats_test_func2 +------------------ + +(1 row) + +COMMIT; +-- rolled back transaction +BEGIN; +SELECT stats_test_func2(); + stats_test_func2 +------------------ + +(1 row) + +ROLLBACK; -- ensure pending stats are flushed SELECT pg_stat_force_next_flush(); pg_stat_force_next_flush @@ -146,6 +250,270 @@ FROM prevstats AS pr; (1 row) COMMIT; +-- check stats are dropped (happens synchronously) +SELECT pg_stat_get_live_tuples(:drop_stats_test_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +DROP TABLE drop_stats_test; +SELECT pg_stat_get_live_tuples(:drop_stats_test_oid); + pg_stat_get_live_tuples +------------------------- + 0 +(1 row) + +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 0 +(1 row) + +-- check that rollback protects against having stats dropped +-- and that local modifcations don't pose a problem +SELECT pg_stat_get_live_tuples(:drop_stats_test_xact_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +SELECT pg_stat_get_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_tuples_inserted +----------------------------- + 1 +(1 row) + +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 0 +(1 row) + +BEGIN; +INSERT INTO drop_stats_test_xact DEFAULT VALUES; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 1 +(1 row) + +DROP TABLE drop_stats_test_xact; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 0 +(1 row) + +ROLLBACK; +SELECT pg_stat_get_live_tuples(:drop_stats_test_xact_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +SELECT pg_stat_get_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_tuples_inserted +----------------------------- + 1 +(1 row) + +-- transactional drop (Cannot use pg_stat_get_xact again, timing +-- dependent whether already flushed) +SELECT pg_stat_get_live_tuples(:drop_stats_test_xact_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +SELECT pg_stat_get_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_tuples_inserted +----------------------------- + 1 +(1 row) + +BEGIN; +INSERT INTO drop_stats_test_xact DEFAULT VALUES; +DROP TABLE drop_stats_test_xact; +COMMIT; +SELECT pg_stat_get_live_tuples(:drop_stats_test_xact_oid); + pg_stat_get_live_tuples +------------------------- + 0 +(1 row) + +SELECT pg_stat_get_tuples_inserted(:drop_stats_test_xact_oid); + pg_stat_get_tuples_inserted +----------------------------- + 0 +(1 row) + +-- savepoint rollback (2 levels) +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +BEGIN; +INSERT INTO drop_stats_test_subxact DEFAULT VALUES; +SAVEPOINT sp1; +INSERT INTO drop_stats_test_subxact DEFAULT VALUES; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_subxact_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 2 +(1 row) + +SAVEPOINT sp2; +DROP TABLE drop_stats_test_subxact; +ROLLBACK TO SAVEPOINT sp2; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_subxact_oid); + pg_stat_get_xact_tuples_inserted +---------------------------------- + 2 +(1 row) + +COMMIT; +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +-- savepoint rolback (1 level) +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +BEGIN; +SAVEPOINT sp1; +DROP TABLE drop_stats_test_subxact; +SAVEPOINT sp2; +ROLLBACK TO SAVEPOINT sp1; +COMMIT; +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +-- and now actually drop +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + pg_stat_get_live_tuples +------------------------- + 1 +(1 row) + +BEGIN; +SAVEPOINT sp1; +DROP TABLE drop_stats_test_subxact; +SAVEPOINT sp2; +RELEASE SAVEPOINT sp1; +COMMIT; +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + pg_stat_get_live_tuples +------------------------- + 0 +(1 row) + +----- +-- continuation of track function tests +----- +-- check stats were collected +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; + funcname | calls +------------------+------- + stats_test_func1 | 2 +(1 row) + +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func2_oid; + funcname | calls +------------------+------- + stats_test_func2 | 4 +(1 row) + +-- check that a rolled back drop function stats leaves stats alive +BEGIN; +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; + funcname | calls +------------------+------- + stats_test_func1 | 2 +(1 row) + +DROP FUNCTION stats_test_func1(); +-- shouldn't be visible via view +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; + funcname | calls +----------+------- +(0 rows) + +-- but still via oid access +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); + pg_stat_get_function_calls +---------------------------- + 2 +(1 row) + +ROLLBACK; +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; + funcname | calls +------------------+------- + stats_test_func1 | 2 +(1 row) + +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); + pg_stat_get_function_calls +---------------------------- + 2 +(1 row) + +-- check that function dropped in main transaction leaves no stats behind +BEGIN; +DROP FUNCTION stats_test_func1(); +COMMIT; +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; + funcname | calls +----------+------- +(0 rows) + +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); + pg_stat_get_function_calls +---------------------------- + +(1 row) + +-- check that function dropped in a subtransaction leaves no stats behind +BEGIN; +SELECT stats_test_func2(); + stats_test_func2 +------------------ + +(1 row) + +SAVEPOINT a; +SELECT stats_test_func2(); + stats_test_func2 +------------------ + +(1 row) + +SAVEPOINT b; +DROP FUNCTION stats_test_func2(); +COMMIT; +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func2_oid; + funcname | calls +----------+------- +(0 rows) + +SELECT pg_stat_get_function_calls(:stats_test_func2_oid); + pg_stat_get_function_calls +---------------------------- + +(1 row) + DROP TABLE trunc_stats_test, trunc_stats_test1, trunc_stats_test2, trunc_stats_test3, trunc_stats_test4; DROP TABLE prevstats; -- ensure that stats accessors handle NULL input correctly diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 1087b2c14f4..103e11483d2 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -123,6 +123,9 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests +# +# The stats test resets stats, so nothing else needing stats access can be in +# this group. # ---------- test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression memoize stats diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql index d49e0aeef85..ae0d8e17c59 100644 --- a/src/test/regress/sql/stats.sql +++ b/src/test/regress/sql/stats.sql @@ -13,6 +13,8 @@ SET enable_seqscan TO on; SET enable_indexscan TO on; -- for the moment, we don't want index-only scans here SET enable_indexonlyscan TO off; +-- not enabled by default, but we want to test it... +SET track_functions TO 'all'; -- save counters BEGIN; @@ -93,11 +95,63 @@ SET enable_bitmapscan TO off; SELECT count(*) FROM tenk2 WHERE unique1 = 1; RESET enable_bitmapscan; +-- Check that stats for tables are dropped. For that we need to access stats +-- by oid after the DROP TABLE. Save oids. +CREATE TABLE drop_stats_test(); +INSERT INTO drop_stats_test DEFAULT VALUES; +SELECT 'drop_stats_test'::regclass::oid AS drop_stats_test_oid \gset + +CREATE TABLE drop_stats_test_xact(); +INSERT INTO drop_stats_test_xact DEFAULT VALUES; +SELECT 'drop_stats_test_xact'::regclass::oid AS drop_stats_test_xact_oid \gset + +CREATE TABLE drop_stats_test_subxact(); +INSERT INTO drop_stats_test_subxact DEFAULT VALUES; +SELECT 'drop_stats_test_subxact'::regclass::oid AS drop_stats_test_subxact_oid \gset + + +-- Basic testing for track_functions +CREATE FUNCTION stats_test_func1() RETURNS VOID LANGUAGE plpgsql AS $$BEGIN END;$$; +SELECT 'stats_test_func1()'::regprocedure::oid AS stats_test_func1_oid \gset +CREATE FUNCTION stats_test_func2() RETURNS VOID LANGUAGE plpgsql AS $$BEGIN END;$$; +SELECT 'stats_test_func2()'::regprocedure::oid AS stats_test_func2_oid \gset + +-- Basic test that stats are accumulated +BEGIN; +SET LOCAL stats_fetch_consistency = none; +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); +SELECT pg_stat_get_xact_function_calls(:stats_test_func1_oid); +SELECT stats_test_func1(); +SELECT pg_stat_get_xact_function_calls(:stats_test_func1_oid); +SELECT stats_test_func1(); +SELECT pg_stat_get_xact_function_calls(:stats_test_func1_oid); +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); +COMMIT; + +-- Verify that function stats are not transactional (displayed after +-- pg_stat_force_next_flush() below) + +-- rolled back savepoint in committing transaction +BEGIN; +SELECT stats_test_func2(); +SAVEPOINT foo; +SELECT stats_test_func2(); +ROLLBACK TO SAVEPOINT foo; +SELECT pg_stat_get_xact_function_calls(:stats_test_func2_oid); +SELECT stats_test_func2(); +COMMIT; + +-- rolled back transaction +BEGIN; +SELECT stats_test_func2(); +ROLLBACK; + -- ensure pending stats are flushed SELECT pg_stat_force_next_flush(); -- check effects BEGIN; + SET LOCAL stats_fetch_consistency = snapshot; SELECT relname, n_tup_ins, n_tup_upd, n_tup_del, n_live_tup, n_dead_tup @@ -121,6 +175,110 @@ FROM prevstats AS pr; COMMIT; + +-- check stats are dropped (happens synchronously) +SELECT pg_stat_get_live_tuples(:drop_stats_test_oid); +DROP TABLE drop_stats_test; +SELECT pg_stat_get_live_tuples(:drop_stats_test_oid); +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_oid); + +-- check that rollback protects against having stats dropped +-- and that local modifcations don't pose a problem +SELECT pg_stat_get_live_tuples(:drop_stats_test_xact_oid); +SELECT pg_stat_get_tuples_inserted(:drop_stats_test_xact_oid); +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_xact_oid); +BEGIN; +INSERT INTO drop_stats_test_xact DEFAULT VALUES; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_xact_oid); +DROP TABLE drop_stats_test_xact; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_xact_oid); +ROLLBACK; +SELECT pg_stat_get_live_tuples(:drop_stats_test_xact_oid); +SELECT pg_stat_get_tuples_inserted(:drop_stats_test_xact_oid); + +-- transactional drop (Cannot use pg_stat_get_xact again, timing +-- dependent whether already flushed) +SELECT pg_stat_get_live_tuples(:drop_stats_test_xact_oid); +SELECT pg_stat_get_tuples_inserted(:drop_stats_test_xact_oid); +BEGIN; +INSERT INTO drop_stats_test_xact DEFAULT VALUES; +DROP TABLE drop_stats_test_xact; +COMMIT; +SELECT pg_stat_get_live_tuples(:drop_stats_test_xact_oid); +SELECT pg_stat_get_tuples_inserted(:drop_stats_test_xact_oid); + +-- savepoint rollback (2 levels) +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); +BEGIN; +INSERT INTO drop_stats_test_subxact DEFAULT VALUES; +SAVEPOINT sp1; +INSERT INTO drop_stats_test_subxact DEFAULT VALUES; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_subxact_oid); +SAVEPOINT sp2; +DROP TABLE drop_stats_test_subxact; +ROLLBACK TO SAVEPOINT sp2; +SELECT pg_stat_get_xact_tuples_inserted(:drop_stats_test_subxact_oid); +COMMIT; +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + +-- savepoint rolback (1 level) +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); +BEGIN; +SAVEPOINT sp1; +DROP TABLE drop_stats_test_subxact; +SAVEPOINT sp2; +ROLLBACK TO SAVEPOINT sp1; +COMMIT; +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + +-- and now actually drop +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); +BEGIN; +SAVEPOINT sp1; +DROP TABLE drop_stats_test_subxact; +SAVEPOINT sp2; +RELEASE SAVEPOINT sp1; +COMMIT; +SELECT pg_stat_get_live_tuples(:drop_stats_test_subxact_oid); + +----- +-- continuation of track function tests +----- + +-- check stats were collected +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func2_oid; + +-- check that a rolled back drop function stats leaves stats alive +BEGIN; +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; +DROP FUNCTION stats_test_func1(); +-- shouldn't be visible via view +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; +-- but still via oid access +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); +ROLLBACK; +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); + +-- check that function dropped in main transaction leaves no stats behind +BEGIN; +DROP FUNCTION stats_test_func1(); +COMMIT; +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func1_oid; +SELECT pg_stat_get_function_calls(:stats_test_func1_oid); + +-- check that function dropped in a subtransaction leaves no stats behind +BEGIN; +SELECT stats_test_func2(); +SAVEPOINT a; +SELECT stats_test_func2(); +SAVEPOINT b; +DROP FUNCTION stats_test_func2(); +COMMIT; +SELECT funcname, calls FROM pg_stat_user_functions WHERE funcid = :stats_test_func2_oid; +SELECT pg_stat_get_function_calls(:stats_test_func2_oid); + DROP TABLE trunc_stats_test, trunc_stats_test1, trunc_stats_test2, trunc_stats_test3, trunc_stats_test4; DROP TABLE prevstats; -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0019-pgstat-test-test-stats-interactions-with-streami.patch" From b10f0532667621a006015945919ca98940525aff Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Mon, 21 Mar 2022 12:58:51 -0700 Subject: [PATCH v68 19/31] pgstat: test: test stats interactions with streaming replication. Author: Melanie Plageman Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- .../recovery/t/030_stats_cleanup_replica.pl | 228 ++++++++++++++++++ 1 file changed, 228 insertions(+) create mode 100644 src/test/recovery/t/030_stats_cleanup_replica.pl diff --git a/src/test/recovery/t/030_stats_cleanup_replica.pl b/src/test/recovery/t/030_stats_cleanup_replica.pl new file mode 100644 index 00000000000..ea79e990395 --- /dev/null +++ b/src/test/recovery/t/030_stats_cleanup_replica.pl @@ -0,0 +1,228 @@ +# Copyright (c) 2021-2022, PostgreSQL Global Development Group + +# Tests that statistics are removed from a physical replica after being dropped +# on the primary + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +# Initialize primary node +my $node_primary = PostgreSQL::Test::Cluster->new('primary'); +# A specific role is created to perform some tests related to replication, +# and it needs proper authentication configuration. +$node_primary->init( + allows_streaming => 1, + auth_extra => [ '--create-role', 'repl_role' ]); +$node_primary->start; +my $backup_name = 'my_backup'; + +# Set track_functions to all on primary +$node_primary->append_conf('postgresql.conf', "track_functions = 'all'"); +$node_primary->reload; + +# Take backup +$node_primary->backup($backup_name); + +# Create streaming standby linking to primary +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node_primary, $backup_name, + has_streaming => 1); +$node_standby->start; + +sub populate_standby_stats +{ + my ($node_primary, $node_standby, $connect_db, $schema) = @_; + # Create table on primary + $node_primary->safe_psql($connect_db, + "CREATE TABLE $schema.drop_tab_test1 AS SELECT generate_series(1,100) AS a" + ); + + # Create function on primary + $node_primary->safe_psql($connect_db, + "CREATE FUNCTION $schema.drop_func_test1() RETURNS VOID AS 'select 2;' LANGUAGE SQL IMMUTABLE" + ); + + # Wait for catchup + my $primary_lsn = $node_primary->lsn('write'); + $node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + + # Get database oid + my $dboid = $node_standby->safe_psql($connect_db, + "SELECT oid FROM pg_database WHERE datname = '$connect_db'"); + + # Get table oid + my $tableoid = $node_standby->safe_psql($connect_db, + "SELECT '$schema.drop_tab_test1'::regclass::oid"); + + # Do scan on standby + $node_standby->safe_psql($connect_db, + "SELECT * FROM $schema.drop_tab_test1"); + + # Get function oid + my $funcoid = $node_standby->safe_psql($connect_db, + "SELECT '$schema.drop_func_test1()'::regprocedure::oid"); + + # Call function on standby + $node_standby->safe_psql($connect_db, "SELECT $schema.drop_func_test1()"); + + # Force flush of stats on standby + $node_standby->safe_psql($connect_db, + "SELECT pg_stat_force_next_flush()"); + + return ($dboid, $tableoid, $funcoid); +} + +sub drop_function_by_oid +{ + my ($node_primary, $connect_db, $funcoid) = @_; + + # Get function name from returned oid + my $func_name = $node_primary->safe_psql($connect_db, + "SELECT '$funcoid'::regprocedure"); + + # Drop function on primary + $node_primary->safe_psql($connect_db, "DROP FUNCTION $func_name"); +} + +sub drop_table_by_oid +{ + my ($node_primary, $connect_db, $tableoid) = @_; + + # Get table name from returned oid + my $table_name = + $node_primary->safe_psql($connect_db, "SELECT '$tableoid'::regclass"); + + # Drop table on primary + $node_primary->safe_psql($connect_db, "DROP TABLE $table_name"); +} + +sub test_standby_func_tab_stats_status +{ + local $Test::Builder::Level = $Test::Builder::Level + 1; + my ($node_standby, $connect_db, $dboid, $tableoid, $funcoid, $present) = + @_; + + is( $node_standby->safe_psql( + $connect_db, + "SELECT pg_stat_exists_stat('relation', $dboid, $tableoid)"), + $present, + "Check that table stats exist is '$present' on standby"); + + is( $node_standby->safe_psql( + $connect_db, + "SELECT pg_stat_exists_stat('function', $dboid, $funcoid)"), + $present, + "Check that function stats exist is '$present' on standby"); + + return; +} + +sub test_standby_db_stats_status +{ + local $Test::Builder::Level = $Test::Builder::Level + 1; + my ($node_standby, $connect_db, $dboid, $present) = @_; + + is( $node_standby->safe_psql( + $connect_db, "SELECT pg_stat_exists_stat('database', $dboid, 0)"), + $present, + "Check that db stats exist is '$present' on standby"); +} + +# Test that stats are cleaned up on standby after dropping table or function + +# Populate test objects +my ($dboid, $tableoid, $funcoid) = + populate_standby_stats($node_primary, $node_standby, 'postgres', 'public'); + +# Test that the stats are present +test_standby_func_tab_stats_status($node_standby, 'postgres', + $dboid, $tableoid, $funcoid, 't'); + +# Drop test objects +drop_table_by_oid($node_primary, 'postgres', $tableoid); + +drop_function_by_oid($node_primary, 'postgres', $funcoid); + +# Wait for catchup +my $primary_lsn = $node_primary->lsn('write'); +$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + +# Force flush of stats on standby +$node_standby->safe_psql('postgres', "SELECT pg_stat_force_next_flush()"); + +# Check table and function stats removed from standby +test_standby_func_tab_stats_status($node_standby, 'postgres', + $dboid, $tableoid, $funcoid, 'f'); + +# Check that stats are cleaned up on standby after dropping schema + +# Create schema +$node_primary->safe_psql('postgres', "CREATE SCHEMA drop_schema_test1"); + +# Wait for catchup +$primary_lsn = $node_primary->lsn('write'); +$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + +# Populate test objects +($dboid, $tableoid, $funcoid) = populate_standby_stats($node_primary, + $node_standby, 'postgres', 'drop_schema_test1'); + +# Test that the stats are present +test_standby_func_tab_stats_status($node_standby, 'postgres', + $dboid, $tableoid, $funcoid, 't'); + +# Drop schema +$node_primary->safe_psql('postgres', "DROP SCHEMA drop_schema_test1 CASCADE"); + +# Wait for catchup +$primary_lsn = $node_primary->lsn('write'); +$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + +# Force flush of stats on standby +$node_standby->safe_psql('postgres', "SELECT pg_stat_force_next_flush()"); + +# Check table and function stats removed from standby +test_standby_func_tab_stats_status($node_standby, 'postgres', + $dboid, $tableoid, $funcoid, 'f'); + +# Test that stats are cleaned up on standby after dropping database + +# Create the database +$node_primary->safe_psql('postgres', "CREATE DATABASE test"); + +# Wait for catchup +$primary_lsn = $node_primary->lsn('write'); +$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + +# Populate test objects +($dboid, $tableoid, $funcoid) = + populate_standby_stats($node_primary, $node_standby, 'test', 'public'); + +# Test that the stats are present +test_standby_func_tab_stats_status($node_standby, 'test', + $dboid, $tableoid, $funcoid, 't'); + +test_standby_db_stats_status($node_standby, 'test', $dboid, 't'); + +# Drop db 'test' on primary +$node_primary->safe_psql('postgres', "DROP DATABASE test"); + +# Wait for catchup +$primary_lsn = $node_primary->lsn('write'); +$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + +# Force flush of stats on standby +$node_standby->safe_psql('postgres', "SELECT pg_stat_force_next_flush()"); + +# Test that the stats were cleaned up on standby +# Note that this connects to 'postgres' but provides the dboid of dropped db +# 'test' which was returned by previous routine +test_standby_func_tab_stats_status($node_standby, 'postgres', + $dboid, $tableoid, $funcoid, 'f'); + +test_standby_db_stats_status($node_standby, 'postgres', $dboid, 'f'); + +done_testing(); -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0020-pgstat-test-stats-handling-of-restarts-including.patch" From 95e4430d2aae61126b6d4ba619b2da55548a4e18 Mon Sep 17 00:00:00 2001 From: Melanie Plageman Date: Sun, 20 Mar 2022 16:33:40 -0400 Subject: [PATCH v68 20/31] pgstat: test: stats handling of restarts, including crashes. Stats should not be restored if the db crashed. Also, ensure invalid stats files are handled gracefully. Author: Melanie Plageman Author: Andres Freund Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- src/test/recovery/t/029_stats_restart.pl | 305 +++++++++++++++++++++++ 1 file changed, 305 insertions(+) create mode 100644 src/test/recovery/t/029_stats_restart.pl diff --git a/src/test/recovery/t/029_stats_restart.pl b/src/test/recovery/t/029_stats_restart.pl new file mode 100644 index 00000000000..55534b3a68d --- /dev/null +++ b/src/test/recovery/t/029_stats_restart.pl @@ -0,0 +1,305 @@ +# Copyright (c) 2021-2022, PostgreSQL Global Development Group + +# Tests statistics handling around restarts, including handling of crashes and +# invalid stats files, as well as restorting stats after "normal" restarts. + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use File::Copy; + +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init(allows_streaming => 1); +$node->start; +$node->append_conf('postgresql.conf', "track_functions = 'all'"); +$node->reload; + +my $connect_db = 'postgres'; +my $db_under_test = 'test'; + +$node->safe_psql($connect_db, "CREATE DATABASE $db_under_test"); + +# Create table in test db +$node->safe_psql($db_under_test, + "CREATE TABLE tab_stats_crash_discard_test1 AS SELECT generate_series(1,100) AS a" +); + +# Create function in test db +$node->safe_psql($db_under_test, + "CREATE FUNCTION func_stats_crash_discard1() RETURNS VOID AS 'select 2;' LANGUAGE SQL IMMUTABLE" +); + +# collect object oids +my $dboid = $node->safe_psql($db_under_test, + "SELECT oid FROM pg_database WHERE datname = '$db_under_test'"); +my $funcoid = $node->safe_psql($db_under_test, + "SELECT 'func_stats_crash_discard1()'::regprocedure::oid"); +my $tableoid = $node->safe_psql($db_under_test, + "SELECT 'tab_stats_crash_discard_test1'::regclass::oid"); + +# generate stats and flush them +trigger_funcrel_stat(); + +# verify stats objects exist +is(exists_stat('database', $dboid, 0), 't', "Check that db stat exists."); +is(exists_stat('function', $dboid, $funcoid), + 't', "Check that function stat exists."); +is(exists_stat('relation', $dboid, $tableoid), + 't', "Check that relation stat exists."); + +# Regular shutdown +$node->stop(); + +# Backup stats files +my $statsfile = $PostgreSQL::Test::Utils::tmp_check . '/' . "discard_stats1"; +ok(!-f "$statsfile", "Backup statsfile cannot already exist"); + +my $datadir = $node->data_dir(); +my $og_stats = $datadir . '/' . "pg_stat" . '/' . "pgstat.stat"; +ok(-f "$og_stats", "Origin stats file must exist"); +copy($og_stats, $statsfile) or die "Copy failed: $!"; + +# Start the server +$node->start; + +is(exists_stat('database', $dboid, 0), 't', "Check that db stat exists."); +is(exists_stat('function', $dboid, $funcoid), + 't', "Check that function stat exists."); +is(exists_stat('relation', $dboid, $tableoid), + 't', "Check that relation stat exists."); + +# Fast shutdown +$node->stop('immediate'); + +ok(!-f "$og_stats", "No stats file should exist after immediate shutdown."); + +# Copy the old stats back to test we discard stats after crash restart +copy($statsfile, $og_stats) or die "Copy failed: $!"; + +# Start the server +$node->start; + +# Stats should have been discarded +is(exists_stat('database', $dboid, 0), + 'f', "Check that db stat does not exist."); +is(exists_stat('function', $dboid, $funcoid), + 'f', "Check that function stat does not exist."); +is(exists_stat('relation', $dboid, $tableoid), + 'f', "Check that relation stat does not exist."); + +# Get rid of backup statsfile +unlink $statsfile or die "cannot unlink $statsfile $!"; + + +# generate new stats and flush them +trigger_funcrel_stat(); + +is(exists_stat('database', $dboid, 0), 't', "Check that db stat exists."); +is(exists_stat('function', $dboid, $funcoid), + 't', "Check that function stat exists."); +is(exists_stat('relation', $dboid, $tableoid), + 't', "Check that relation stat exists."); + +# Regular shutdown +$node->stop(); + +sub overwrite_file +{ + my ($filename, $str) = @_; + open my $fh, ">", $filename + or die "could not write \"$filename\": $!"; + print $fh $str; + close $fh; + return; +} + +overwrite_file($og_stats, "ZZZZZZZZZZZZZ"); + +# Normal startup and no issues despite invalid stats file +$node->start; + +# No stats present due to invalid stats file +is(exists_stat('database', $dboid, 0), + 'f', "Check that db stat does not exist."); +is(exists_stat('function', $dboid, $funcoid), + 'f', "Check that function stat does not exist."); +is(exists_stat('relation', $dboid, $tableoid), + 'f', "Check that relation stat does not exist."); + + +## Checks related to stats persistency around restarts and resets + +# Ensure enough checkpoints to protect against races for test after reset, +# even on very slow machines. +$node->safe_psql($connect_db, "CHECKPOINT; CHECKPOINT;"); + + +## check checkpoint and wal stats are incremented due to restart + +my $ckpt_start = checkpoint_stats(); +my $wal_start = wal_stats(); +$node->restart; + +my $sect = "post restart"; +my $ckpt_restart = checkpoint_stats(); +my $wal_restart = wal_stats(); + +cmp_ok( + $ckpt_start->{count}, '<', + $ckpt_restart->{count}, + "$sect: increased checkpoint count"); +cmp_ok( + $wal_start->{records}, '<', + $wal_restart->{records}, + "$sect: increased wal record count"); +cmp_ok($wal_start->{bytes}, '<', $wal_restart->{bytes}, + "$sect: increased wal bytes"); +is( $ckpt_start->{reset}, + $ckpt_restart->{reset}, + "$sect: checkpoint stats_reset equal"); +is($wal_start->{reset}, $wal_restart->{reset}, + "$sect: wal stats_reset equal"); + + +## Check that checkpoint stats are reset, WAL stats aren't affected + +$node->safe_psql($connect_db, "SELECT pg_stat_reset_shared('bgwriter')"); + +$sect = "post ckpt reset"; +my $ckpt_reset = checkpoint_stats(); +my $wal_ckpt_reset = wal_stats(); + +cmp_ok($ckpt_restart->{count}, + '>', $ckpt_reset->{count}, "$sect: checkpoint count smaller"); +cmp_ok($ckpt_start->{reset}, 'lt', $ckpt_reset->{reset}, + "$sect: stats_reset newer"); + +cmp_ok( + $wal_restart->{records}, + '<=', + $wal_ckpt_reset->{records}, + "$sect: wal record count not affected by reset"); +is( $wal_start->{reset}, + $wal_ckpt_reset->{reset}, + "$sect: wal stats_reset equal"); + + +## check that checkpoint stats stay reset after restart + +$node->restart; + +$sect = "post ckpt reset & restart"; +my $ckpt_restart_reset = checkpoint_stats(); +my $wal_restart2 = wal_stats(); + +# made sure above there's enough checkpoints that this will be stable even on slow machines +cmp_ok( + $ckpt_restart_reset->{count}, + '<', + $ckpt_restart->{count}, + "$sect: checkpoint still reset"); +is($ckpt_restart_reset->{reset}, + $ckpt_reset->{reset}, "$sect: stats_reset same"); + +cmp_ok( + $wal_ckpt_reset->{records}, + '<', + $wal_restart2->{records}, + "$sect: increased wal record count"); +cmp_ok( + $wal_ckpt_reset->{bytes}, + '<', + $wal_restart2->{bytes}, + "$sect: increased wal bytes"); +is( $wal_start->{reset}, + $wal_restart2->{reset}, + "$sect: wal stats_reset equal"); + + +## check WAL stats stay reset + +$node->safe_psql($connect_db, "SELECT pg_stat_reset_shared('wal')"); + +$sect = "post wal reset"; +my $wal_reset = wal_stats(); + +cmp_ok( + $wal_reset->{records}, '<', + $wal_restart2->{records}, + "$sect: smaller record count"); +cmp_ok( + $wal_reset->{bytes}, '<', + $wal_restart2->{bytes}, + "$sect: smaller bytes"); +cmp_ok( + $wal_reset->{reset}, 'gt', + $wal_restart2->{reset}, + "$sect: newer stats_reset"); + +$node->restart; + +$sect = "post wal reset & restart"; +my $wal_reset_restart = wal_stats(); + +# enough WAL generated during prior tests and initdb to make this not racy +cmp_ok( + $wal_reset_restart->{records}, + '<', + $wal_restart2->{records}, + "$sect: smaller record count"); +cmp_ok( + $wal_reset->{bytes}, '<', + $wal_restart2->{bytes}, + "$sect: smaller bytes"); +cmp_ok( + $wal_reset->{reset}, 'gt', + $wal_restart2->{reset}, + "$sect: newer stats_reset"); + + +$node->stop; +done_testing(); + +sub trigger_funcrel_stat +{ + $node->safe_psql( + $db_under_test, q[ + SELECT * FROM tab_stats_crash_discard_test1; + SELECT func_stats_crash_discard1(); + SELECT pg_stat_force_next_flush();]); +} + +sub exists_stat +{ + my ($kind, $dboid, $objoid) = @_; + + return $node->safe_psql($connect_db, + "SELECT pg_stat_exists_stat('$kind', $dboid, $objoid)"); +} + +sub checkpoint_stats +{ + my %results; + + $results{count} = $node->safe_psql($connect_db, + "SELECT checkpoints_timed + checkpoints_req FROM pg_stat_bgwriter"); + $results{reset} = $node->safe_psql($connect_db, + "SELECT stats_reset FROM pg_stat_bgwriter"); + + return \%results; +} + +sub wal_stats +{ + my %results; + $results{records} = + $node->safe_psql($connect_db, "SELECT wal_records FROM pg_stat_wal"); + $results{bytes} = + $node->safe_psql($connect_db, "SELECT wal_bytes FROM pg_stat_wal"); + $results{reset} = + $node->safe_psql($connect_db, "SELECT stats_reset FROM pg_stat_wal"); + + return \%results; +} -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0021-pgstat-test-subscriber-stats-reset-and-drop.patch" From 1cf700e399c2a5b7c9d7a2489534165c8d1b63ac Mon Sep 17 00:00:00 2001 From: Melanie Plageman Date: Mon, 21 Mar 2022 14:52:55 -0400 Subject: [PATCH v68 21/31] pgstat: test: subscriber stats reset and drop Author: Melanie Plageman Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- src/test/subscription/t/026_stats.pl | 305 +++++++++++++++++++++------ 1 file changed, 240 insertions(+), 65 deletions(-) diff --git a/src/test/subscription/t/026_stats.pl b/src/test/subscription/t/026_stats.pl index a42ea3170ec..7aa9a9254de 100644 --- a/src/test/subscription/t/026_stats.pl +++ b/src/test/subscription/t/026_stats.pl @@ -18,83 +18,258 @@ my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); $node_subscriber->init(allows_streaming => 'logical'); $node_subscriber->start; -# Initial table setup on both publisher and subscriber. On subscriber we -# create the same tables but with primary keys. Also, insert some data that -# will conflict with the data replicated from publisher later. -$node_publisher->safe_psql( - 'postgres', - qq[ -BEGIN; -CREATE TABLE test_tab1 (a int); -INSERT INTO test_tab1 VALUES (1); -COMMIT; -]); -$node_subscriber->safe_psql( - 'postgres', - qq[ -BEGIN; -CREATE TABLE test_tab1 (a int primary key); -INSERT INTO test_tab1 VALUES (1); -COMMIT; -]); -# Setup publication. -my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; -$node_publisher->safe_psql('postgres', - "CREATE PUBLICATION tap_pub FOR TABLE test_tab1;"); +sub create_sub_pub_w_errors +{ + my ($node_publisher, $node_subscriber, $db, $table_name) = @_; + # Initial table setup on both publisher and subscriber. On subscriber we + # create the same tables but with primary keys. Also, insert some data that + # will conflict with the data replicated from publisher later. + $node_publisher->safe_psql( + $db, + qq[ + BEGIN; + CREATE TABLE $table_name(a int); + INSERT INTO $table_name VALUES (1); + COMMIT; + ]); + $node_subscriber->safe_psql( + $db, + qq[ + BEGIN; + CREATE TABLE $table_name(a int primary key); + INSERT INTO $table_name VALUES (1); + COMMIT; + ]); + + # Set up publication. + my $pub_name = $table_name . '_pub'; + my $publisher_connstr = $node_publisher->connstr . qq( dbname=$db); + + $node_publisher->safe_psql($db, + qq(CREATE PUBLICATION $pub_name FOR TABLE $table_name)); + + # Create subscription. The tablesync for table on subscription will enter into + # infinite error loop due to violating the unique constraint. + my $sub_name = $table_name . '_sub'; + $node_subscriber->safe_psql($db, + qq(CREATE SUBSCRIPTION $sub_name CONNECTION '$publisher_connstr' PUBLICATION $pub_name) + ); + + $node_publisher->wait_for_catchup($sub_name); + + # Wait for the tablesync error to be reported. + $node_subscriber->poll_query_until( + $db, + qq[ + SELECT sync_error_count > 0 + FROM pg_stat_subscription_stats + WHERE subname = '$sub_name' + ]) + or die + qq(Timed out while waiting for tablesync errors for subscription '$sub_name'); + + # Truncate test_tab1 so that tablesync worker can continue. + $node_subscriber->safe_psql($db, qq(TRUNCATE $table_name)); + + # Wait for initial tablesync to finish. + $node_subscriber->poll_query_until( + $db, + qq[ + SELECT count(1) = 1 FROM pg_subscription_rel + WHERE srrelid = '$table_name'::regclass AND srsubstate in ('r', 's') + ]) + or die + qq(Timed out while waiting for subscriber to synchronize data for table '$table_name'.); + + # Check test table on the subscriber has one row. + my $result = + $node_subscriber->safe_psql($db, qq(SELECT a FROM $table_name)); + is($result, qq(1), qq(Check that table '$table_name' now has 1 row.)); + + # Insert data to test table on the publisher, raising an error on the + # subscriber due to violation of the unique constraint on test table. + $node_publisher->safe_psql($db, qq(INSERT INTO $table_name VALUES (1))); + + # Wait for the apply error to be reported. + $node_subscriber->poll_query_until( + $db, + qq[ + SELECT apply_error_count > 0 + FROM pg_stat_subscription_stats + WHERE subname = '$sub_name' + ]) + or die + qq(Timed out while waiting for apply error for subscription '$sub_name'); + + # Truncate test table so that apply worker can continue. + $node_subscriber->safe_psql($db, qq(TRUNCATE $table_name)); + + return ($pub_name, $sub_name); +} + +my $db = 'postgres'; # There shouldn't be any subscription errors before starting logical replication. -my $result = $node_subscriber->safe_psql('postgres', - "SELECT count(1) FROM pg_stat_subscription_stats"); -is($result, qq(0), 'check no subscription error'); - -# Create subscription. The tablesync for test_tab1 on tap_sub will enter into -# infinite error loop due to violating the unique constraint. -$node_subscriber->safe_psql('postgres', - "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub;" +my $result = $node_subscriber->safe_psql($db, + qq(SELECT count(1) FROM pg_stat_subscription_stats)); +is($result, qq(0), + 'Check that there are no subscription errors before starting logical replication.' ); -$node_publisher->wait_for_catchup('tap_sub'); +# Create the publication and subscription with sync and apply errors +my $table1_name = 'test_tab1'; +my ($pub1_name, $sub1_name) = + create_sub_pub_w_errors($node_publisher, $node_subscriber, $db, + $table1_name); -# Wait for the tablesync error to be reported. -$node_subscriber->poll_query_until( - 'postgres', - qq[ -SELECT sync_error_count > 0 -FROM pg_stat_subscription_stats -WHERE subname = 'tap_sub' -]) or die "Timed out while waiting for tablesync error"; +# Apply and Sync errors are > 0 and reset timestamp is NULL +is( $node_subscriber->safe_psql( + $db, + qq(SELECT apply_error_count > 0, + sync_error_count > 0, + stats_reset IS NULL + FROM pg_stat_subscription_stats + WHERE subname = '$sub1_name') + ), + qq(t|t|t), + qq(Check that apply errors and sync errors are both > 0 and stats_reset is NULL for subscription '$sub1_name'.) +); -# Truncate test_tab1 so that tablesync worker can continue. -$node_subscriber->safe_psql('postgres', "TRUNCATE test_tab1;"); +# Reset a single subscription +$node_subscriber->safe_psql($db, + qq(SELECT pg_stat_reset_subscription_stats((SELECT subid FROM pg_stat_subscription_stats WHERE subname = '$sub1_name'))) +); -# Wait for initial tablesync for test_tab1 to finish. -$node_subscriber->poll_query_until( - 'postgres', - qq[ -SELECT count(1) = 1 FROM pg_subscription_rel -WHERE srrelid = 'test_tab1'::regclass AND srsubstate in ('r', 's') -]) or die "Timed out while waiting for subscriber to synchronize data"; +# Apply and Sync errors are 0 and stats reset is not NULL +is( $node_subscriber->safe_psql( + $db, + qq(SELECT apply_error_count = 0, + sync_error_count = 0, + stats_reset IS NOT NULL + FROM pg_stat_subscription_stats + WHERE subname = '$sub1_name') + ), + qq(t|t|t), + qq(Confirm that apply errors and sync errors are both 0 and stats_reset is not NULL after reset for subscription '$sub1_name'.) +); -# Check test_tab1 on the subscriber has one row. -$result = $node_subscriber->safe_psql('postgres', "SELECT a FROM test_tab1"); -is($result, qq(1), 'check the table has now row'); +# Get reset timestamp +my $reset_time1 = $node_subscriber->safe_psql($db, + qq(SELECT stats_reset FROM pg_stat_subscription_stats WHERE subname = '$sub1_name') +); -# Insert data to test_tab1 on the publisher, raising an error on the subscriber -# due to violation of the unique constraint on test_tab1. -$node_publisher->safe_psql('postgres', "INSERT INTO test_tab1 VALUES (1)"); +# Reset single sub again +$node_subscriber->safe_psql( + $db, + qq(SELECT pg_stat_reset_subscription_stats((SELECT subid FROM + pg_stat_subscription_stats WHERE subname = '$sub1_name'))) +); -# Wait for the apply error to be reported. -$node_subscriber->poll_query_until( - 'postgres', - qq[ -SELECT apply_error_count > 0 -FROM pg_stat_subscription_stats -WHERE subname = 'tap_sub' -]) or die "Timed out while waiting for apply error"; +# check reset timestamp is newer after reset +is( $node_subscriber->safe_psql( + $db, + qq(SELECT stats_reset > '$reset_time1'::timestamptz FROM + pg_stat_subscription_stats WHERE subname = '$sub1_name') + ), + qq(t), + qq(Check reset timestamp for '$sub1_name' is newer after second reset.)); + +# Make second subscription and publication +my $table2_name = 'test_tab2'; +my ($pub2_name, $sub2_name) = + create_sub_pub_w_errors($node_publisher, $node_subscriber, $db, + $table2_name); + +# Apply and Sync errors are > 0 and reset timestamp is NULL +is( $node_subscriber->safe_psql( + $db, + qq(SELECT apply_error_count > 0, + sync_error_count > 0, + stats_reset IS NULL + FROM pg_stat_subscription_stats + WHERE subname = '$sub2_name') + ), + qq(t|t|t), + qq(Confirm that apply errors and sync errors are both > 0 and stats_reset is NULL for sub '$sub2_name'.) +); + +# Reset all subscriptions +$node_subscriber->safe_psql($db, + qq(SELECT pg_stat_reset_subscription_stats(NULL))); + +# Apply and Sync errors are 0 and stats reset is not NULL +is( $node_subscriber->safe_psql( + $db, + qq(SELECT apply_error_count = 0, + sync_error_count = 0, + stats_reset IS NOT NULL + FROM pg_stat_subscription_stats + WHERE subname = '$sub1_name') + ), + qq(t|t|t), + qq(Confirm that apply errors and sync errors are both 0 and stats_reset is not NULL for sub '$sub1_name' after reset.) +); + +is( $node_subscriber->safe_psql( + $db, + qq(SELECT apply_error_count = 0, + sync_error_count = 0, + stats_reset IS NOT NULL + FROM pg_stat_subscription_stats + WHERE subname = '$sub2_name') + ), + qq(t|t|t), + qq(Confirm that apply errors and sync errors are both 0 and stats_reset is not NULL for sub '$sub2_name' after reset.) +); + +# Get reset timestamp 1 +$reset_time1 = $node_subscriber->safe_psql($db, + qq(SELECT stats_reset FROM pg_stat_subscription_stats WHERE subname = '$sub1_name') +); + +# Get reset timestamp 2 +my $reset_time2 = $node_subscriber->safe_psql($db, + qq(SELECT stats_reset FROM pg_stat_subscription_stats WHERE subname = '$sub2_name') +); + +# Reset all subscriptions +$node_subscriber->safe_psql($db, + qq(SELECT pg_stat_reset_subscription_stats(NULL))); + +# check reset timestamp for sub1 is newer after reset +is( $node_subscriber->safe_psql( + $db, + qq(SELECT stats_reset > '$reset_time1'::timestamptz FROM + pg_stat_subscription_stats WHERE subname = '$sub1_name') + ), + qq(t), + qq(Confirm that reset timestamp for '$sub1_name' is newer after second reset.) +); + +# check reset timestamp for sub2 is newer after reset +is( $node_subscriber->safe_psql( + $db, + qq(SELECT stats_reset > '$reset_time2'::timestamptz FROM + pg_stat_subscription_stats WHERE subname = '$sub2_name') + ), + qq(t), + qq(Confirm that reset timestamp for '$sub2_name' is newer after second reset.) +); + +# Get subscription 1 oid +my $sub1_oid = $node_subscriber->safe_psql($db, + qq(SELECT oid FROM pg_subscription WHERE subname = '$sub1_name')); + +# Drop subscription 1 +$node_subscriber->safe_psql($db, qq(DROP SUBSCRIPTION $sub1_name)); + +# Subscription stats for sub1 should be gone +is( $node_subscriber->safe_psql( + $db, qq(SELECT pg_stat_exists_stat('subscription', 0, $sub1_oid))), + qq(f), + qq(Subscription stats for subscription '$sub1_name' should be removed.)); -# Truncate test_tab1 so that apply worker can continue. -$node_subscriber->safe_psql('postgres', "TRUNCATE test_tab1;"); $node_subscriber->stop('fast'); $node_publisher->stop('fast'); -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0022-pgstat-test-resetting-of-stats.patch" From f3de6afb3115e15ed69b9e06516abf3dcc856aa1 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sun, 3 Apr 2022 15:17:16 -0700 Subject: [PATCH v68 22/31] pgstat: test: resetting of stats. --- src/test/recovery/t/006_logical_decoding.pl | 63 ++++++ src/test/regress/expected/stats.out | 212 ++++++++++++++++++++ src/test/regress/sql/stats.sql | 110 ++++++++++ 3 files changed, 385 insertions(+) diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl index 9cec2792fc2..3ccced2ea24 100644 --- a/src/test/recovery/t/006_logical_decoding.pl +++ b/src/test/recovery/t/006_logical_decoding.pl @@ -200,6 +200,69 @@ chomp($logical_restart_lsn_post); ok(($logical_restart_lsn_pre cmp $logical_restart_lsn_post) == 0, "logical slot advance persists across restarts"); +my $stats_test_slot1 = 'test_slot'; +my $stats_test_slot2 = 'logical_slot'; + +# Test that reset works for pg_stat_replication_slots + +# Stats exist for stats test slot 1 +is($node_primary->safe_psql( + 'postgres', + qq(SELECT total_bytes > 0, stats_reset IS NULL FROM pg_stat_replication_slots WHERE slot_name = '$stats_test_slot1') +), qq(t|t), qq(Total bytes is > 0 and stats_reset is NULL for slot '$stats_test_slot1'.)); + +# Do reset of stats for stats test slot 1 +$node_primary->safe_psql( + 'postgres', + qq(SELECT pg_stat_reset_replication_slot('$stats_test_slot1')) +); + +# Get reset value after reset +my $reset1 = $node_primary->safe_psql( + 'postgres', + qq(SELECT stats_reset FROM pg_stat_replication_slots WHERE slot_name = '$stats_test_slot1') +); + +# Do reset again +$node_primary->safe_psql( + 'postgres', + qq(SELECT pg_stat_reset_replication_slot('$stats_test_slot1')) +); + +is($node_primary->safe_psql( + 'postgres', + qq(SELECT stats_reset > '$reset1'::timestamptz, total_bytes = 0 FROM pg_stat_replication_slots WHERE slot_name = '$stats_test_slot1') +), qq(t|t), qq(Check that reset timestamp is later after the second reset of stats for slot '$stats_test_slot1' and confirm total_bytes was set to 0.)); + +# Check that test slot 2 has NULL in reset timestamp +is($node_primary->safe_psql( + 'postgres', + qq(SELECT stats_reset IS NULL FROM pg_stat_replication_slots WHERE slot_name = '$stats_test_slot2') +), qq(t), qq(Stats_reset is NULL for slot '$stats_test_slot2' before reset.)); + +# Get reset value again for test slot 1 +$reset1 = $node_primary->safe_psql( + 'postgres', + qq(SELECT stats_reset FROM pg_stat_replication_slots WHERE slot_name = '$stats_test_slot1') +); + +# Reset stats for all replication slots +$node_primary->safe_psql( + 'postgres', + qq(SELECT pg_stat_reset_replication_slot(NULL)) +); + +# Check that test slot 2 reset timestamp is no longer NULL after reset +is($node_primary->safe_psql( + 'postgres', + qq(SELECT stats_reset IS NOT NULL FROM pg_stat_replication_slots WHERE slot_name = '$stats_test_slot2') +), qq(t), qq(Stats_reset is not NULL for slot '$stats_test_slot2' after reset all.)); + +is($node_primary->safe_psql( + 'postgres', + qq(SELECT stats_reset > '$reset1'::timestamptz FROM pg_stat_replication_slots WHERE slot_name = '$stats_test_slot1') +), qq(t), qq(Check that reset timestamp is later after resetting stats for slot '$stats_test_slot1' again.)); + # done with the node $node_primary->stop; diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out index 890b5d7e5bc..24550d87a03 100644 --- a/src/test/regress/expected/stats.out +++ b/src/test/regress/expected/stats.out @@ -529,4 +529,216 @@ SELECT pg_stat_get_subscription_stats(NULL); (1 row) +----- +-- Test that various stats views are being properly populated +----- +-- Test that sessions is incremented when a new session is started in pg_stat_database +SELECT sessions AS db_stat_sessions FROM pg_stat_database WHERE datname = (SELECT current_database()) \gset +\c +SELECT sessions > :db_stat_sessions FROM pg_stat_database WHERE datname = (SELECT current_database()); + ?column? +---------- + t +(1 row) + +-- Test pg_stat_bgwriter checkpointer-related stat +SELECT checkpoints_req AS rqst_ckpts_before FROM pg_stat_bgwriter \gset +-- Test pg_stat_wal +SELECT wal_bytes AS wal_bytes_before FROM pg_stat_wal \gset +CREATE TABLE test_stats_temp(a int); +INSERT INTO test_stats_temp SELECT 1 FROM generate_series(1,1000)i; +-- Checkpoint twice: The checkpointer reports stats after reporting completion +-- of the checkpoint. But after a second checkpoint we'll see at least the +-- results of the first. +CHECKPOINT; +CHECKPOINT; +SELECT checkpoints_req > :rqst_ckpts_before FROM pg_stat_bgwriter; + ?column? +---------- + t +(1 row) + +SELECT wal_bytes > :wal_bytes_before FROM pg_stat_wal; + ?column? +---------- + t +(1 row) + +----- +-- Test that resetting stats works for reset timestamp +----- +SELECT stats_reset AS slru_commit_ts_reset_ts FROM pg_stat_slru WHERE name = 'CommitTs' \gset +SELECT stats_reset AS slru_notify_reset_ts FROM pg_stat_slru WHERE name = 'Notify' \gset +-- Test that reset_slru with a specified SLRU works. +SELECT pg_stat_reset_slru('CommitTs'); + pg_stat_reset_slru +-------------------- + +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT stats_reset > :'slru_commit_ts_reset_ts'::timestamptz FROM pg_stat_slru WHERE name = 'CommitTs'; + ?column? +---------- + t +(1 row) + +SELECT stats_reset AS slru_commit_ts_reset_ts FROM pg_stat_slru WHERE name = 'CommitTs' \gset +-- Test that multiple SLRUs are reset when no specific SLRU provided to reset function +SELECT pg_stat_reset_slru(NULL); + pg_stat_reset_slru +-------------------- + +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT stats_reset > :'slru_commit_ts_reset_ts'::timestamptz FROM pg_stat_slru WHERE name = 'CommitTs'; + ?column? +---------- + t +(1 row) + +SELECT stats_reset > :'slru_notify_reset_ts'::timestamptz FROM pg_stat_slru WHERE name = 'Notify'; + ?column? +---------- + t +(1 row) + +SELECT stats_reset AS archiver_reset_ts FROM pg_stat_archiver \gset +-- Test that reset_shared with archiver specified as the stats type works +SELECT pg_stat_reset_shared('archiver'); + pg_stat_reset_shared +---------------------- + +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT stats_reset > :'archiver_reset_ts'::timestamptz FROM pg_stat_archiver; + ?column? +---------- + t +(1 row) + +SELECT stats_reset AS archiver_reset_ts FROM pg_stat_archiver \gset +SELECT stats_reset AS bgwriter_reset_ts FROM pg_stat_bgwriter \gset +-- Test that reset_shared with bgwriter specified as the stats type works +SELECT pg_stat_reset_shared('bgwriter'); + pg_stat_reset_shared +---------------------- + +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT stats_reset > :'bgwriter_reset_ts'::timestamptz FROM pg_stat_bgwriter; + ?column? +---------- + t +(1 row) + +SELECT stats_reset AS bgwriter_reset_ts FROM pg_stat_bgwriter \gset +SELECT stats_reset AS wal_reset_ts FROM pg_stat_wal \gset +-- Test that reset_shared with wal specified as the stats type works +SELECT pg_stat_reset_shared('wal'); + pg_stat_reset_shared +---------------------- + +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal; + ?column? +---------- + t +(1 row) + +SELECT stats_reset AS wal_reset_ts FROM pg_stat_wal \gset +-- Test that reset_shared with no specified stats type resets all of them +SELECT pg_stat_reset_shared(NULL); + pg_stat_reset_shared +---------------------- + +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT stats_reset > :'archiver_reset_ts'::timestamptz FROM pg_stat_archiver; + ?column? +---------- + f +(1 row) + +SELECT stats_reset > :'bgwriter_reset_ts'::timestamptz FROM pg_stat_bgwriter; + ?column? +---------- + f +(1 row) + +SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal; + ?column? +---------- + f +(1 row) + +-- Test that reset works for pg_stat_database +-- Since pg_stat_database stats_reset starts out as NULL, reset it once first so we have something to compare it to +SELECT pg_stat_reset(); + pg_stat_reset +--------------- + +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT stats_reset AS db_reset_ts FROM pg_stat_database WHERE datname = (SELECT current_database()) \gset +SELECT pg_stat_reset(); + pg_stat_reset +--------------- + +(1 row) + +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT stats_reset > :'db_reset_ts'::timestamptz FROM pg_stat_database WHERE datname = (SELECT current_database()); + ?column? +---------- + t +(1 row) + -- End of Stats Test diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql index ae0d8e17c59..8b7282183d0 100644 --- a/src/test/regress/sql/stats.sql +++ b/src/test/regress/sql/stats.sql @@ -286,5 +286,115 @@ DROP TABLE prevstats; SELECT pg_stat_get_replication_slot(NULL); SELECT pg_stat_get_subscription_stats(NULL); +----- +-- Test that various stats views are being properly populated +----- + +-- Test that sessions is incremented when a new session is started in pg_stat_database +SELECT sessions AS db_stat_sessions FROM pg_stat_database WHERE datname = (SELECT current_database()) \gset +\c +SELECT sessions > :db_stat_sessions FROM pg_stat_database WHERE datname = (SELECT current_database()); + +-- Test pg_stat_bgwriter checkpointer-related stat + +SELECT checkpoints_req AS rqst_ckpts_before FROM pg_stat_bgwriter \gset + +-- Test pg_stat_wal + +SELECT wal_bytes AS wal_bytes_before FROM pg_stat_wal \gset + +CREATE TABLE test_stats_temp(a int); +INSERT INTO test_stats_temp SELECT 1 FROM generate_series(1,1000)i; + +-- Checkpoint twice: The checkpointer reports stats after reporting completion +-- of the checkpoint. But after a second checkpoint we'll see at least the +-- results of the first. +CHECKPOINT; +CHECKPOINT; + +SELECT checkpoints_req > :rqst_ckpts_before FROM pg_stat_bgwriter; +SELECT wal_bytes > :wal_bytes_before FROM pg_stat_wal; + +----- +-- Test that resetting stats works for reset timestamp +----- + +SELECT stats_reset AS slru_commit_ts_reset_ts FROM pg_stat_slru WHERE name = 'CommitTs' \gset + +SELECT stats_reset AS slru_notify_reset_ts FROM pg_stat_slru WHERE name = 'Notify' \gset + +-- Test that reset_slru with a specified SLRU works. +SELECT pg_stat_reset_slru('CommitTs'); + +SELECT pg_stat_force_next_flush(); + +SELECT stats_reset > :'slru_commit_ts_reset_ts'::timestamptz FROM pg_stat_slru WHERE name = 'CommitTs'; + +SELECT stats_reset AS slru_commit_ts_reset_ts FROM pg_stat_slru WHERE name = 'CommitTs' \gset + +-- Test that multiple SLRUs are reset when no specific SLRU provided to reset function +SELECT pg_stat_reset_slru(NULL); + +SELECT pg_stat_force_next_flush(); + +SELECT stats_reset > :'slru_commit_ts_reset_ts'::timestamptz FROM pg_stat_slru WHERE name = 'CommitTs'; +SELECT stats_reset > :'slru_notify_reset_ts'::timestamptz FROM pg_stat_slru WHERE name = 'Notify'; + +SELECT stats_reset AS archiver_reset_ts FROM pg_stat_archiver \gset + +-- Test that reset_shared with archiver specified as the stats type works +SELECT pg_stat_reset_shared('archiver'); + +SELECT pg_stat_force_next_flush(); + +SELECT stats_reset > :'archiver_reset_ts'::timestamptz FROM pg_stat_archiver; + +SELECT stats_reset AS archiver_reset_ts FROM pg_stat_archiver \gset + +SELECT stats_reset AS bgwriter_reset_ts FROM pg_stat_bgwriter \gset + +-- Test that reset_shared with bgwriter specified as the stats type works +SELECT pg_stat_reset_shared('bgwriter'); + +SELECT pg_stat_force_next_flush(); + +SELECT stats_reset > :'bgwriter_reset_ts'::timestamptz FROM pg_stat_bgwriter; + +SELECT stats_reset AS bgwriter_reset_ts FROM pg_stat_bgwriter \gset + +SELECT stats_reset AS wal_reset_ts FROM pg_stat_wal \gset + +-- Test that reset_shared with wal specified as the stats type works +SELECT pg_stat_reset_shared('wal'); + +SELECT pg_stat_force_next_flush(); + +SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal; + +SELECT stats_reset AS wal_reset_ts FROM pg_stat_wal \gset + +-- Test that reset_shared with no specified stats type resets all of them +SELECT pg_stat_reset_shared(NULL); + +SELECT pg_stat_force_next_flush(); + +SELECT stats_reset > :'archiver_reset_ts'::timestamptz FROM pg_stat_archiver; +SELECT stats_reset > :'bgwriter_reset_ts'::timestamptz FROM pg_stat_bgwriter; +SELECT stats_reset > :'wal_reset_ts'::timestamptz FROM pg_stat_wal; + +-- Test that reset works for pg_stat_database + +-- Since pg_stat_database stats_reset starts out as NULL, reset it once first so we have something to compare it to +SELECT pg_stat_reset(); + +SELECT pg_stat_force_next_flush(); + +SELECT stats_reset AS db_reset_ts FROM pg_stat_database WHERE datname = (SELECT current_database()) \gset + +SELECT pg_stat_reset(); + +SELECT pg_stat_force_next_flush(); + +SELECT stats_reset > :'db_reset_ts'::timestamptz FROM pg_stat_database WHERE datname = (SELECT current_database()); -- End of Stats Test -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0023-pgstat-test-recovery-conflicts.patch" From b4fad5da282a899d2edcbb2341800f28826bb3cc Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Thu, 31 Mar 2022 17:30:39 -0700 Subject: [PATCH v68 23/31] pgstat: test: recovery conflicts. Author: Melanie Plageman --- src/test/recovery/t/031_recovery_conflict.pl | 378 +++++++++++++++++++ 1 file changed, 378 insertions(+) create mode 100644 src/test/recovery/t/031_recovery_conflict.pl diff --git a/src/test/recovery/t/031_recovery_conflict.pl b/src/test/recovery/t/031_recovery_conflict.pl new file mode 100644 index 00000000000..4d25c281f82 --- /dev/null +++ b/src/test/recovery/t/031_recovery_conflict.pl @@ -0,0 +1,378 @@ +# Copyright (c) 2021-2022, PostgreSQL Global Development Group + +# Test that connections to a hot standby are correctly canceled when a recovery conflict is detected +# Also, test that statistics in pg_stat_database_conflicts are populated correctly + +# TODO: add a test for deadlock recovery conflicts. + +# recovery conflicts +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $psql_timeout = IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default); +# Initialize primary node +my $node_primary = PostgreSQL::Test::Cluster->new('primary'); +$node_primary->init( + allows_streaming => 1, + auth_extra => [ '--create-role', 'repl_role' ]); + +my $tablespace1 = "test_recovery_conflict_tblspc"; + +$node_primary->append_conf( + 'postgresql.conf', qq[ +allow_in_place_tablespaces = on +log_temp_files = 0 + +# wait some to test the wait paths as well, but not long for obvious reasons +max_standby_streaming_delay = 50ms + +temp_tablespaces = $tablespace1 +# Some of the recovery conflict logging code only gets exercised after +# deadlock_timeout. The test doesn't rely on that additional output, but it's +# nice to get some minimal coverage of that code. +log_recovery_conflict_waits = on +deadlock_timeout = 10ms +]); +$node_primary->start; + +my $backup_name = 'my_backup'; + +$node_primary->safe_psql('postgres', + qq[CREATE TABLESPACE $tablespace1 LOCATION '']); + +# Take backup +$node_primary->backup($backup_name); + +# Create streaming standby linking to primary +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node_primary, $backup_name, + has_streaming => 1); +$node_standby->start; + +my $test_db = "test_db"; + +# Make a new database for these tests so that it can also be used for the +# database recovery conflict test. +$node_primary->safe_psql( + 'postgres', + qq[ + CREATE DATABASE $test_db; + ]); + +# Make table for first few recovery conflict types under test +my $table1 = "test_recovery_conflict_table1"; + +$node_primary->safe_psql($test_db, qq[CREATE TABLE ${table1}(a int, b int);]); + +$node_primary->safe_psql($test_db, + qq[INSERT INTO $table1 SELECT i % 3, 0 FROM generate_series(1,20) i]); + +# Wait for catchup so database and tables exists when establishing connection +my $primary_lsn = $node_primary->lsn('write'); +$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + +# Make a psql connection on the standby +my %psql_standby = ('stdin' => '', 'stdout' => '', 'stderr' => ''); +$psql_standby{run} = IPC::Run::start( + [ 'psql', '-XA', '-f', '-', '-d', $node_standby->connstr($test_db) ], + '<', + \$psql_standby{stdin}, + '>', + \$psql_standby{stdout}, + '2>', + \$psql_standby{stderr}, + $psql_timeout); + + +my $nconflicts = 0; + +# RECOVERY CONFLICT 1: Buffer pin conflict +$nconflicts++; + +# Aborted INSERT on primary +$node_primary->safe_psql( + $test_db, + qq[ + BEGIN; + INSERT INTO $table1 VALUES (1,0); + ROLLBACK; + ]); + +$primary_lsn = $node_primary->lsn('write'); +$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + +my $cursor1 = "test_recovery_conflict_cursor"; + +# DECLARE and use a cursor on standby, causing buffer with the only block of +# the relation to be pinned on the standby +$psql_standby{stdin} .= qq[ + BEGIN; + DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1; + FETCH FORWARD FROM $cursor1; + ]; +# FETCH FORWARD should have returned a 0 since all values of b in the table are +# 0 +ok( pump_until( + $psql_standby{run}, $psql_timeout, + \$psql_standby{stdout}, qr/^0$/m,), + "cursor"); + +# Reset the streams to avoid false positive test results in later tests +($psql_standby{stdin}, $psql_standby{stderr}, $psql_standby{stdout}) = + ('', '', ''); + +# Get log location before taking action to cause disconnect of connections on +# standby +my $log_location = -s $node_standby->logfile; + +# VACUUM on the primary +$node_primary->safe_psql($test_db, qq[VACUUM $table1;]); + +# Wait for catchup. Existing connection will be terminated before replay is +# finished, so waiting for catchup ensures that there is no race between +# encountering the recovery conflict which causes the disconnect and checking +# the logfile for the terminated connection. +$primary_lsn = $node_primary->lsn('write'); +$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + +cmp_ok( + $node_standby->wait_for_log( + qr/User was holding shared buffer pin for too long/, + $log_location), + '>', + $log_location, + 'Logfile should contain details of a terminated connection due to the buffer pin recovery conflict.' +); +reconnect_and_clear(\%psql_standby, $psql_timeout); + +is( $node_standby->safe_psql( + $test_db, + qq[SELECT confl_bufferpin FROM pg_stat_database_conflicts WHERE datname='$test_db';] + ), + 1, + 'Check that buffer pin conflict is captured in stats on standby.'); + +# RECOVERY CONFLICT 2: Snapshot conflict +$nconflicts++; + +# INSERT new data +$node_primary->safe_psql($test_db, + qq[INSERT INTO $table1 SELECT i, 0 FROM generate_series(1,20) i]); + +# Wait for catchup +$primary_lsn = $node_primary->lsn('write'); +$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + +# DECLARE and FETCH from cursor on the standby +$psql_standby{stdin} .= qq[ + BEGIN; + DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1; + FETCH FORWARD FROM $cursor1; + ]; +ok( pump_until( + $psql_standby{run}, $psql_timeout, + \$psql_standby{stdout}, qr/^0$/m,), + "cursor"); + +# Do some HOT updates +$node_primary->safe_psql( + $test_db, + qq[ + UPDATE $table1 SET a = a + 1 WHERE a > 2; + ]); + +# Update log location +$log_location = -s $node_standby->logfile; + +# VACUUM, pruning those dead tuples +$node_primary->safe_psql( + $test_db, + qq[ + VACUUM $table1; + ]); + +# Wait for attempted replay of PRUNE records +$primary_lsn = $node_primary->lsn('write'); +$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + +cmp_ok( + $node_standby->wait_for_log( + qr/User query might have needed to see row versions that must be removed/, + $log_location), + '>', + $log_location, + 'Logfile should contain details of a terminated connection due to snapshot recovery conflict.' +); +reconnect_and_clear(\%psql_standby, $psql_timeout); + +is( $node_standby->safe_psql( + $test_db, + qq[SELECT confl_snapshot FROM pg_stat_database_conflicts WHERE datname='$test_db';] + ), + 1, + 'Check that snapshot conflict was captured in stats on standby.'); + + +# RECOVERY CONFLICT 3: Lock conflict +$nconflicts++; + +# DECLARE and FETCH from cursor on the standby +$psql_standby{stdin} .= qq[ + BEGIN; + DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1; + FETCH FORWARD FROM $cursor1; + ]; +ok( pump_until( + $psql_standby{run}, $psql_timeout, + \$psql_standby{stdout}, qr/^0$/m,), + "cursor"); + +# Update log location +$log_location = -s $node_standby->logfile; + +# DROP TABLE containing block which standby has in a pinned buffer +$node_primary->safe_psql( + $test_db, + qq[ + DROP TABLE $table1; + ]); + +# Wait for catchup +$primary_lsn = $node_primary->lsn('write'); +$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + +cmp_ok( + $node_standby->wait_for_log( + qr/User was holding a relation lock for too long/, + $log_location), + '>', + $log_location, + 'Logfile should contain details of a terminated connection due to lock recovery conflict.' +); +reconnect_and_clear(\%psql_standby, $psql_timeout); + +is( $node_standby->safe_psql( + $test_db, + qq[SELECT confl_lock FROM pg_stat_database_conflicts WHERE datname='$test_db';] + ), + 1, + 'Check that lock conflict is captured in stats on standby.'); + + +# RECOVERY CONFLICT 4: Tablespace conflict +$nconflicts++; + +# DECLARE a cursor for a query which, with sufficiently low work_mem, will +# spill tuples into temp files in the temporary tablespace created during +# setup. +$psql_standby{stdin} .= qq[ + BEGIN; + SET work_mem = '64kB'; + DECLARE $cursor1 CURSOR FOR + SELECT count(*) FROM generate_series(1,6000); + FETCH FORWARD FROM $cursor1; + ]; +ok( pump_until( + $psql_standby{run}, $psql_timeout, + \$psql_standby{stdout}, qr/^\(1 row\)$/m,), + "cursor"); + +# Update log location +$log_location = -s $node_standby->logfile; + +# Drop the tablespace currently containing spill files for the query on the +# standby +$node_primary->safe_psql( + $test_db, + qq[ + DROP TABLESPACE $tablespace1; + ]); + +# Wait for catchup +$primary_lsn = $node_primary->lsn('write'); +$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + +cmp_ok( + $node_standby->wait_for_log( + qr/User was or might have been using tablespace that must be dropped/, + $log_location), + '>', + $log_location, + 'Logfile should contain details of a terminated connection due to tablespace recovery conflict.' +); +reconnect_and_clear(\%psql_standby, $psql_timeout); + +is( $node_standby->safe_psql( + $test_db, + qq[SELECT confl_tablespace FROM pg_stat_database_conflicts WHERE datname='$test_db';] + ), + 1, + 'Check that tablespace conflict was captured in stats on standby.'); + +is( $node_standby->safe_psql( + $test_db, + qq[SELECT conflicts FROM pg_stat_database WHERE datname='$test_db';]), + $nconflicts, + qq[Check that all $nconflicts recovery conflicts have been captured in pg_stat_database.] +); + + +# RECOVERY CONFLICT 5: Database conflict +$nconflicts++; + +# Update log location +$log_location = -s $node_standby->logfile; + +$node_primary->safe_psql( + 'postgres', + qq[ + DROP DATABASE $test_db; + ]); + +# Wait for catchup +$primary_lsn = $node_primary->lsn('write'); +$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn); + +cmp_ok( + $node_standby->wait_for_log( + qr/User was connected to a database that must be dropped/, + $log_location), + '>', + $log_location, + 'Logfile should contain details of a terminated connection due to database recovery conflict.' +); + +# explicitly shut down psql instances gracefully - to avoid hangs +# or worse on windows +$psql_standby{stdin} .= "\\q\n"; +$psql_standby{run}->finish; + +$node_standby->stop(); +$node_primary->stop(); + +sub reconnect_and_clear +{ + # Note that all of the entries in the hash passed in must match those in + # this subroutine exactly + my ($psql, $psql_timeout) = @_; + + $$psql{stdin} .= "\\q\n"; + $$psql{run}->finish; + + # Run + $$psql{run}->run(); + + # Reset streams + ($$psql{stdin}, $$psql{stderr}, $$psql{stdout}) = ('', '', ''); + + # Run query to ensure connection has been re-established + $$psql{stdin} .= qq[ + SELECT 1; + ]; + die unless pump_until($$psql{run}, $psql_timeout, \$$psql{stdout}, qr/^1$/m,); +} + +done_testing(); -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0024-pgstat-test-extend-replication-slot-stat-tests.patch" From 106ce6952bc35e82523e65505db14a3623e631f1 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sun, 27 Mar 2022 22:05:53 -0700 Subject: [PATCH v68 24/31] pgstat: test: extend replication slot stat tests --- contrib/test_decoding/expected/stats.out | 117 +++++++++++++++++------ contrib/test_decoding/sql/stats.sql | 32 +++++-- 2 files changed, 111 insertions(+), 38 deletions(-) diff --git a/contrib/test_decoding/expected/stats.out b/contrib/test_decoding/expected/stats.out index 5dd1926176c..800c727d6e1 100644 --- a/contrib/test_decoding/expected/stats.out +++ b/contrib/test_decoding/expected/stats.out @@ -1,6 +1,9 @@ -- predictability SET synchronous_commit = on; -SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot_stats', 'test_decoding'); +SELECT 'init' FROM + pg_create_logical_replication_slot('regression_slot_stats1', 'test_decoding') s1, + pg_create_logical_replication_slot('regression_slot_stats2', 'test_decoding') s2, + pg_create_logical_replication_slot('regression_slot_stats3', 'test_decoding') s3; ?column? ---------- init @@ -10,7 +13,19 @@ CREATE TABLE stats_test(data text); -- non-spilled xact SET logical_decoding_work_mem to '64MB'; INSERT INTO stats_test values(1); -SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot_stats', NULL, NULL, 'skip-empty-xacts', '1'); +SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot_stats1', NULL, NULL, 'skip-empty-xacts', '1'); + count +------- + 3 +(1 row) + +SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot_stats2', NULL, NULL, 'skip-empty-xacts', '1'); + count +------- + 3 +(1 row) + +SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot_stats3', NULL, NULL, 'skip-empty-xacts', '1'); count ------- 3 @@ -22,31 +37,65 @@ SELECT pg_stat_force_next_flush(); (1 row) -SELECT slot_name, spill_txns = 0 AS spill_txns, spill_count = 0 AS spill_count, total_txns > 0 AS total_txns, total_bytes > 0 AS total_bytes FROM pg_stat_replication_slots; - slot_name | spill_txns | spill_count | total_txns | total_bytes ------------------------+------------+-------------+------------+------------- - regression_slot_stats | t | t | t | t -(1 row) +SELECT slot_name, spill_txns = 0 AS spill_txns, spill_count = 0 AS spill_count, total_txns > 0 AS total_txns, total_bytes > 0 AS total_bytes FROM pg_stat_replication_slots ORDER BY slot_name; + slot_name | spill_txns | spill_count | total_txns | total_bytes +------------------------+------------+-------------+------------+------------- + regression_slot_stats1 | t | t | t | t + regression_slot_stats2 | t | t | t | t + regression_slot_stats3 | t | t | t | t +(3 rows) RESET logical_decoding_work_mem; --- reset the slot stats -SELECT pg_stat_reset_replication_slot('regression_slot_stats'); +-- reset stats for one slot, others should be unaffected +SELECT pg_stat_reset_replication_slot('regression_slot_stats1'); pg_stat_reset_replication_slot -------------------------------- (1 row) -SELECT slot_name, spill_txns, spill_count, total_txns, total_bytes FROM pg_stat_replication_slots; - slot_name | spill_txns | spill_count | total_txns | total_bytes ------------------------+------------+-------------+------------+------------- - regression_slot_stats | 0 | 0 | 0 | 0 +SELECT slot_name, spill_txns = 0 AS spill_txns, spill_count = 0 AS spill_count, total_txns > 0 AS total_txns, total_bytes > 0 AS total_bytes FROM pg_stat_replication_slots ORDER BY slot_name; + slot_name | spill_txns | spill_count | total_txns | total_bytes +------------------------+------------+-------------+------------+------------- + regression_slot_stats1 | t | t | f | f + regression_slot_stats2 | t | t | t | t + regression_slot_stats3 | t | t | t | t +(3 rows) + +-- reset stats for all slots +SELECT pg_stat_reset_replication_slot(NULL); + pg_stat_reset_replication_slot +-------------------------------- + +(1 row) + +SELECT slot_name, spill_txns = 0 AS spill_txns, spill_count = 0 AS spill_count, total_txns > 0 AS total_txns, total_bytes > 0 AS total_bytes FROM pg_stat_replication_slots ORDER BY slot_name; + slot_name | spill_txns | spill_count | total_txns | total_bytes +------------------------+------------+-------------+------------+------------- + regression_slot_stats1 | t | t | f | f + regression_slot_stats2 | t | t | f | f + regression_slot_stats3 | t | t | f | f +(3 rows) + +-- verify accessing/resetting stats for non-existent slot does something reasonable +SELECT * FROM pg_stat_get_replication_slot('do-not-exist'); + slot_name | spill_txns | spill_count | spill_bytes | stream_txns | stream_count | stream_bytes | total_txns | total_bytes | stats_reset +--------------+------------+-------------+-------------+-------------+--------------+--------------+------------+-------------+------------- + do-not-exist | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | +(1 row) + +SELECT pg_stat_reset_replication_slot('do-not-exist'); +ERROR: replication slot "do-not-exist" does not exist +SELECT * FROM pg_stat_get_replication_slot('do-not-exist'); + slot_name | spill_txns | spill_count | spill_bytes | stream_txns | stream_count | stream_bytes | total_txns | total_bytes | stats_reset +--------------+------------+-------------+-------------+-------------+--------------+--------------+------------+-------------+------------- + do-not-exist | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | (1 row) -- spilling the xact BEGIN; INSERT INTO stats_test SELECT 'serialize-topbig--1:'||g.i FROM generate_series(1, 5000) g(i); COMMIT; -SELECT count(*) FROM pg_logical_slot_peek_changes('regression_slot_stats', NULL, NULL, 'skip-empty-xacts', '1'); +SELECT count(*) FROM pg_logical_slot_peek_changes('regression_slot_stats1', NULL, NULL, 'skip-empty-xacts', '1'); count ------- 5002 @@ -62,32 +111,40 @@ SELECT pg_stat_force_next_flush(); (1 row) SELECT slot_name, spill_txns > 0 AS spill_txns, spill_count > 0 AS spill_count FROM pg_stat_replication_slots; - slot_name | spill_txns | spill_count ------------------------+------------+------------- - regression_slot_stats | t | t -(1 row) + slot_name | spill_txns | spill_count +------------------------+------------+------------- + regression_slot_stats1 | t | t + regression_slot_stats2 | f | f + regression_slot_stats3 | f | f +(3 rows) -- Ensure stats can be repeatedly accessed using the same stats snapshot. See -- https://postgr.es/m/20210317230447.c7uc4g3vbs4wi32i%40alap3.anarazel.de BEGIN; SET LOCAL stats_fetch_consistency = snapshot; SELECT slot_name FROM pg_stat_replication_slots; - slot_name ------------------------ - regression_slot_stats -(1 row) + slot_name +------------------------ + regression_slot_stats1 + regression_slot_stats2 + regression_slot_stats3 +(3 rows) SELECT slot_name FROM pg_stat_replication_slots; - slot_name ------------------------ - regression_slot_stats -(1 row) + slot_name +------------------------ + regression_slot_stats1 + regression_slot_stats2 + regression_slot_stats3 +(3 rows) COMMIT; DROP TABLE stats_test; -SELECT pg_drop_replication_slot('regression_slot_stats'); - pg_drop_replication_slot --------------------------- - +SELECT pg_drop_replication_slot('regression_slot_stats1'), + pg_drop_replication_slot('regression_slot_stats2'), + pg_drop_replication_slot('regression_slot_stats3'); + pg_drop_replication_slot | pg_drop_replication_slot | pg_drop_replication_slot +--------------------------+--------------------------+-------------------------- + | | (1 row) diff --git a/contrib/test_decoding/sql/stats.sql b/contrib/test_decoding/sql/stats.sql index 464b9cd1fe3..3c2d26138d6 100644 --- a/contrib/test_decoding/sql/stats.sql +++ b/contrib/test_decoding/sql/stats.sql @@ -1,27 +1,41 @@ -- predictability SET synchronous_commit = on; -SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot_stats', 'test_decoding'); +SELECT 'init' FROM + pg_create_logical_replication_slot('regression_slot_stats1', 'test_decoding') s1, + pg_create_logical_replication_slot('regression_slot_stats2', 'test_decoding') s2, + pg_create_logical_replication_slot('regression_slot_stats3', 'test_decoding') s3; CREATE TABLE stats_test(data text); -- non-spilled xact SET logical_decoding_work_mem to '64MB'; INSERT INTO stats_test values(1); -SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot_stats', NULL, NULL, 'skip-empty-xacts', '1'); +SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot_stats1', NULL, NULL, 'skip-empty-xacts', '1'); +SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot_stats2', NULL, NULL, 'skip-empty-xacts', '1'); +SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot_stats3', NULL, NULL, 'skip-empty-xacts', '1'); SELECT pg_stat_force_next_flush(); -SELECT slot_name, spill_txns = 0 AS spill_txns, spill_count = 0 AS spill_count, total_txns > 0 AS total_txns, total_bytes > 0 AS total_bytes FROM pg_stat_replication_slots; +SELECT slot_name, spill_txns = 0 AS spill_txns, spill_count = 0 AS spill_count, total_txns > 0 AS total_txns, total_bytes > 0 AS total_bytes FROM pg_stat_replication_slots ORDER BY slot_name; RESET logical_decoding_work_mem; --- reset the slot stats -SELECT pg_stat_reset_replication_slot('regression_slot_stats'); -SELECT slot_name, spill_txns, spill_count, total_txns, total_bytes FROM pg_stat_replication_slots; +-- reset stats for one slot, others should be unaffected +SELECT pg_stat_reset_replication_slot('regression_slot_stats1'); +SELECT slot_name, spill_txns = 0 AS spill_txns, spill_count = 0 AS spill_count, total_txns > 0 AS total_txns, total_bytes > 0 AS total_bytes FROM pg_stat_replication_slots ORDER BY slot_name; + +-- reset stats for all slots +SELECT pg_stat_reset_replication_slot(NULL); +SELECT slot_name, spill_txns = 0 AS spill_txns, spill_count = 0 AS spill_count, total_txns > 0 AS total_txns, total_bytes > 0 AS total_bytes FROM pg_stat_replication_slots ORDER BY slot_name; + +-- verify accessing/resetting stats for non-existent slot does something reasonable +SELECT * FROM pg_stat_get_replication_slot('do-not-exist'); +SELECT pg_stat_reset_replication_slot('do-not-exist'); +SELECT * FROM pg_stat_get_replication_slot('do-not-exist'); -- spilling the xact BEGIN; INSERT INTO stats_test SELECT 'serialize-topbig--1:'||g.i FROM generate_series(1, 5000) g(i); COMMIT; -SELECT count(*) FROM pg_logical_slot_peek_changes('regression_slot_stats', NULL, NULL, 'skip-empty-xacts', '1'); +SELECT count(*) FROM pg_logical_slot_peek_changes('regression_slot_stats1', NULL, NULL, 'skip-empty-xacts', '1'); -- Check stats. We can't test the exact stats count as that can vary if any -- background transaction (say by autovacuum) happens in parallel to the main @@ -38,4 +52,6 @@ SELECT slot_name FROM pg_stat_replication_slots; COMMIT; DROP TABLE stats_test; -SELECT pg_drop_replication_slot('regression_slot_stats'); +SELECT pg_drop_replication_slot('regression_slot_stats1'), + pg_drop_replication_slot('regression_slot_stats2'), + pg_drop_replication_slot('regression_slot_stats3'); -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0025-pgstat-remove-stats_temp_directory.patch" From ac0d3dc25a772cae0a1269c43dfc55e64f2e4fab Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Wed, 30 Mar 2022 08:24:14 -0700 Subject: [PATCH v68 25/31] pgstat: remove stats_temp_directory. With stats now being stored in shared memory, the GUC isn't needed anymore. However, the pg_stat_tmp directory and PG_STAT_TMP_DIR define are kept because pg_stat_statements (and some out-of-core extensions) store data in it. Author: Andres Freund Discussion: https://postgr.es/m/20220330233550.eiwsbearu6xhuqwe@alap3.anarazel.de Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- src/include/pgstat.h | 5 --- src/backend/postmaster/pgstat.c | 13 ------ src/backend/replication/basebackup.c | 36 +--------------- src/backend/utils/misc/guc.c | 41 ------------------- src/backend/utils/misc/postgresql.conf.sample | 1 - src/bin/pg_rewind/filemap.c | 5 +-- .../pg_stat_statements/pg_stat_statements.c | 7 +--- src/test/perl/PostgreSQL/Test/Cluster.pm | 4 -- 8 files changed, 5 insertions(+), 107 deletions(-) diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fae3186c8eb..8234f9fdfb9 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -647,11 +647,6 @@ extern PGDLLIMPORT bool pgstat_track_counts; extern PGDLLIMPORT int pgstat_track_functions; extern PGDLLIMPORT int pgstat_fetch_consistency; -/* No longer used, but will be removed with GUC */ -extern char *pgstat_stat_directory; -extern char *pgstat_stat_tmpname; -extern char *pgstat_stat_filename; - /* * Variables in pgstat_bgwriter.c diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index 8d619d114b2..d0315147bf1 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/postmaster/pgstat.c @@ -129,19 +129,6 @@ bool pgstat_track_counts = false; int pgstat_fetch_consistency = PGSTAT_FETCH_CONSISTENCY_NONE; -/* ---------- - * Built from GUC parameter - * ---------- - */ - -char *pgstat_stat_directory = NULL; - -/* No longer used, but will be removed with GUC */ -char *pgstat_stat_filename = NULL; -char *pgstat_stat_tmpname = NULL; - - - /* ---------- * Local data * diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c index db9fd0a7290..d31b06d885e 100644 --- a/src/backend/replication/basebackup.c +++ b/src/backend/replication/basebackup.c @@ -99,9 +99,6 @@ static int basebackup_read_file(int fd, char *buf, size_t nbytes, off_t offset, /* Was the backup currently in-progress initiated in recovery mode? */ static bool backup_started_in_recovery = false; -/* Relative path of temporary statistics directory */ -static char *statrelpath = NULL; - /* Total number of checksum failures during base backup. */ static long long int total_checksum_failures; @@ -131,9 +128,8 @@ struct exclude_list_item static const char *const excludeDirContents[] = { /* - * Skip temporary statistics files. PG_STAT_TMP_DIR must be skipped even - * when stats_temp_directory is set because PGSS_TEXT_FILE is always - * created there. + * Skip temporary statistics files. PG_STAT_TMP_DIR must be skipped + * because extensions like pg_stat_statements store data there. */ PG_STAT_TMP_DIR, @@ -239,7 +235,6 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) StringInfo labelfile; StringInfo tblspc_map_file; backup_manifest_info manifest; - int datadirpathlen; /* Initial backup state, insofar as we know it now. */ state.tablespaces = NIL; @@ -252,8 +247,6 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) Assert(CurrentResourceOwner == NULL); CurrentResourceOwner = ResourceOwnerCreate(NULL, "base backup"); - datadirpathlen = strlen(DataDir); - backup_started_in_recovery = RecoveryInProgress(); labelfile = makeStringInfo(); @@ -281,18 +274,6 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) ListCell *lc; tablespaceinfo *ti; - /* - * Calculate the relative path of temporary statistics directory in - * order to skip the files which are located in that directory later. - */ - if (is_absolute_path(pgstat_stat_directory) && - strncmp(pgstat_stat_directory, DataDir, datadirpathlen) == 0) - statrelpath = psprintf("./%s", pgstat_stat_directory + datadirpathlen + 1); - else if (strncmp(pgstat_stat_directory, "./", 2) != 0) - statrelpath = psprintf("./%s", pgstat_stat_directory); - else - statrelpath = pgstat_stat_directory; - /* Add a node for the base directory at the end */ ti = palloc0(sizeof(tablespaceinfo)); ti->size = -1; @@ -1312,19 +1293,6 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly, if (excludeFound) continue; - /* - * Exclude contents of directory specified by statrelpath if not set - * to the default (pg_stat_tmp) which is caught in the loop above. - */ - if (statrelpath != NULL && strcmp(pathbuf, statrelpath) == 0) - { - elog(DEBUG1, "contents of directory \"%s\" excluded from backup", statrelpath); - convert_link_to_directory(pathbuf, &statbuf); - size += _tarWriteHeader(sink, pathbuf + basepathlen + 1, NULL, - &statbuf, sizeonly); - continue; - } - /* * We can skip pg_wal, the WAL segments need to be fetched from the * WAL archive anyway. But include it as an empty directory anyway, so diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index bbb156e0dfe..8ff02ef2fb3 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -216,7 +216,6 @@ static bool check_effective_io_concurrency(int *newval, void **extra, GucSource static bool check_maintenance_io_concurrency(int *newval, void **extra, GucSource source); static bool check_huge_page_size(int *newval, void **extra, GucSource source); static bool check_client_connection_check_interval(int *newval, void **extra, GucSource source); -static void assign_pgstat_temp_directory(const char *newval, void *extra); static bool check_application_name(char **newval, void **extra, GucSource source); static void assign_application_name(const char *newval, void *extra); static bool check_cluster_name(char **newval, void **extra, GucSource source); @@ -4559,17 +4558,6 @@ static struct config_string ConfigureNamesString[] = NULL, NULL, NULL }, - { - {"stats_temp_directory", PGC_SIGHUP, STATS_COLLECTOR, - gettext_noop("Writes temporary statistics files to the specified directory."), - NULL, - GUC_SUPERUSER_ONLY - }, - &pgstat_temp_directory, - PG_STAT_TMP_DIR, - check_canonical_path, assign_pgstat_temp_directory, NULL - }, - { {"synchronous_standby_names", PGC_SIGHUP, REPLICATION_PRIMARY, gettext_noop("Number of synchronous standbys and list of names of potential synchronous ones."), @@ -12271,35 +12259,6 @@ check_client_connection_check_interval(int *newval, void **extra, GucSource sour return true; } -static void -assign_pgstat_temp_directory(const char *newval, void *extra) -{ - /* check_canonical_path already canonicalized newval for us */ - char *dname; - char *tname; - char *fname; - - /* directory */ - dname = guc_malloc(ERROR, strlen(newval) + 1); /* runtime dir */ - sprintf(dname, "%s", newval); - - /* global stats */ - tname = guc_malloc(ERROR, strlen(newval) + 12); /* /global.tmp */ - sprintf(tname, "%s/global.tmp", newval); - fname = guc_malloc(ERROR, strlen(newval) + 13); /* /global.stat */ - sprintf(fname, "%s/global.stat", newval); - - if (pgstat_stat_directory) - free(pgstat_stat_directory); - pgstat_stat_directory = dname; - if (pgstat_stat_tmpname) - free(pgstat_stat_tmpname); - pgstat_stat_tmpname = tname; - if (pgstat_stat_filename) - free(pgstat_stat_filename); - pgstat_stat_filename = fname; -} - static bool check_application_name(char **newval, void **extra, GucSource source) { diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 5f9a37bed3b..64e5d11cd69 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -613,7 +613,6 @@ #track_io_timing = off #track_wal_io_timing = off #track_functions = none # none, pl, all -#stats_temp_directory = 'pg_stat_tmp' #stats_fetch_consistency = none diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c index 7211090f471..1a6c67bdd19 100644 --- a/src/bin/pg_rewind/filemap.c +++ b/src/bin/pg_rewind/filemap.c @@ -88,9 +88,8 @@ struct exclude_list_item static const char *excludeDirContents[] = { /* - * Skip temporary statistics files. PG_STAT_TMP_DIR must be skipped even - * when stats_temp_directory is set because PGSS_TEXT_FILE is always - * created there. + * Skip temporary statistics files. PG_STAT_TMP_DIR must be skipped + * because extensions like pg_stat_statements store data there. */ "pg_stat_tmp", /* defined as PG_STAT_TMP_DIR */ diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c index 55786ae84f2..1d6049f2fd8 100644 --- a/contrib/pg_stat_statements/pg_stat_statements.c +++ b/contrib/pg_stat_statements/pg_stat_statements.c @@ -78,12 +78,7 @@ PG_MODULE_MAGIC; #define PGSS_DUMP_FILE PGSTAT_STAT_PERMANENT_DIRECTORY "/pg_stat_statements.stat" /* - * Location of external query text file. We don't keep it in the core - * system's stats_temp_directory. The core system can safely use that GUC - * setting, because the statistics collector temp file paths are set only once - * as part of changing the GUC, but pg_stat_statements has no way of avoiding - * race conditions. Besides, we only expect modest, infrequent I/O for query - * strings, so placing the file on a faster filesystem is not compelling. + * Location of external query text file. */ #define PGSS_TEXT_FILE PG_STAT_TMP_DIR "/pgss_query_texts.stat" diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm index b4ebc999356..50e80ae9640 100644 --- a/src/test/perl/PostgreSQL/Test/Cluster.pm +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm @@ -480,10 +480,6 @@ sub init print $conf PostgreSQL::Test::Utils::slurp_file($ENV{TEMP_CONFIG}) if defined $ENV{TEMP_CONFIG}; - # XXX Neutralize any stats_temp_directory in TEMP_CONFIG. Nodes running - # concurrently must not share a stats_temp_directory. - print $conf "stats_temp_directory = 'pg_stat_tmp'\n"; - if ($params{allows_streaming}) { if ($params{allows_streaming} eq "logical") -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0026-pgstat-rename-STATS_COLLECTOR-GUC-group-to-STATS.patch" From 9a19ae585e83dc75da8890eabf32427d598d82ef Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sun, 3 Apr 2022 14:40:49 -0700 Subject: [PATCH v68 26/31] pgstat: rename STATS_COLLECTOR GUC group to STATS_CUMULATIVE. Author: Andres Freund Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- src/include/utils/guc_tables.h | 2 +- src/backend/utils/misc/guc.c | 18 +++++++++--------- src/backend/utils/misc/postgresql.conf.sample | 2 +- 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/src/include/utils/guc_tables.h b/src/include/utils/guc_tables.h index f7e54a87b7b..1c5b3930a97 100644 --- a/src/include/utils/guc_tables.h +++ b/src/include/utils/guc_tables.h @@ -82,7 +82,7 @@ enum config_group LOGGING_WHAT, PROCESS_TITLE, STATS_MONITORING, - STATS_COLLECTOR, + STATS_CUMULATIVE, AUTOVACUUM, CLIENT_CONN_STATEMENT, CLIENT_CONN_LOCALE, diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 8ff02ef2fb3..3faf35cff61 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -809,8 +809,8 @@ const char *const config_group_names[] = gettext_noop("Reporting and Logging / Process Title"), /* STATS_MONITORING */ gettext_noop("Statistics / Monitoring"), - /* STATS_COLLECTOR */ - gettext_noop("Statistics / Query and Index Statistics Collector"), + /* STATS_CUMULATIVE */ + gettext_noop("Statistics / Cumulative Query and Index Statistics"), /* AUTOVACUUM */ gettext_noop("Autovacuum"), /* CLIENT_CONN_STATEMENT */ @@ -1547,7 +1547,7 @@ static struct config_bool ConfigureNamesBool[] = #endif { - {"track_activities", PGC_SUSET, STATS_COLLECTOR, + {"track_activities", PGC_SUSET, STATS_CUMULATIVE, gettext_noop("Collects information about executing commands."), gettext_noop("Enables the collection of information on the currently " "executing command of each session, along with " @@ -1558,7 +1558,7 @@ static struct config_bool ConfigureNamesBool[] = NULL, NULL, NULL }, { - {"track_counts", PGC_SUSET, STATS_COLLECTOR, + {"track_counts", PGC_SUSET, STATS_CUMULATIVE, gettext_noop("Collects statistics on database activity."), NULL }, @@ -1567,7 +1567,7 @@ static struct config_bool ConfigureNamesBool[] = NULL, NULL, NULL }, { - {"track_io_timing", PGC_SUSET, STATS_COLLECTOR, + {"track_io_timing", PGC_SUSET, STATS_CUMULATIVE, gettext_noop("Collects timing statistics for database I/O activity."), NULL }, @@ -1576,7 +1576,7 @@ static struct config_bool ConfigureNamesBool[] = NULL, NULL, NULL }, { - {"track_wal_io_timing", PGC_SUSET, STATS_COLLECTOR, + {"track_wal_io_timing", PGC_SUSET, STATS_CUMULATIVE, gettext_noop("Collects timing statistics for WAL I/O activity."), NULL }, @@ -3538,7 +3538,7 @@ static struct config_int ConfigureNamesInt[] = }, { - {"track_activity_query_size", PGC_POSTMASTER, STATS_COLLECTOR, + {"track_activity_query_size", PGC_POSTMASTER, STATS_CUMULATIVE, gettext_noop("Sets the size reserved for pg_stat_activity.query, in bytes."), NULL, GUC_UNIT_BYTE @@ -4906,7 +4906,7 @@ static struct config_enum ConfigureNamesEnum[] = }, { - {"track_functions", PGC_SUSET, STATS_COLLECTOR, + {"track_functions", PGC_SUSET, STATS_CUMULATIVE, gettext_noop("Collects function-level statistics on database activity."), NULL }, @@ -4917,7 +4917,7 @@ static struct config_enum ConfigureNamesEnum[] = { - {"stats_fetch_consistency", PGC_USERSET, STATS_COLLECTOR, + {"stats_fetch_consistency", PGC_USERSET, STATS_CUMULATIVE, gettext_noop("Sets the consistency of accesses to statistics data"), NULL }, diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 64e5d11cd69..e75b7d63ea3 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -605,7 +605,7 @@ # STATISTICS #------------------------------------------------------------------------------ -# - Query and Index Statistics Collector - +# - Cumulative Query and Index Statistics - #track_activities = on #track_activity_query_size = 1024 # (change requires restart) -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0027-pgstat-move-pgstat.c-to-utils-activity.patch" From 4d86c7df0960c68b6e7369228a07d95b6f5dab47 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Wed, 16 Mar 2022 13:35:14 -0700 Subject: [PATCH v68 27/31] pgstat: move pgstat.c to utils/activity. Now that pgstat is not related to postmaster anymore, src/backend/postmaster is not a well fitting directory. Author: Andres Freund Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de --- src/backend/postmaster/Makefile | 1 - src/backend/utils/activity/Makefile | 1 + src/backend/{postmaster => utils/activity}/pgstat.c | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) rename src/backend/{postmaster => utils/activity}/pgstat.c (99%) diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile index dbbeac5a82f..3a794e54d60 100644 --- a/src/backend/postmaster/Makefile +++ b/src/backend/postmaster/Makefile @@ -21,7 +21,6 @@ OBJS = \ fork_process.o \ interrupt.o \ pgarch.o \ - pgstat.o \ postmaster.o \ shell_archive.o \ startup.o \ diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile index 690312308f5..a2e8507fd65 100644 --- a/src/backend/utils/activity/Makefile +++ b/src/backend/utils/activity/Makefile @@ -16,6 +16,7 @@ include $(top_builddir)/src/Makefile.global OBJS = \ backend_progress.o \ backend_status.o \ + pgstat.o \ pgstat_archiver.o \ pgstat_bgwriter.o \ pgstat_checkpointer.o \ diff --git a/src/backend/postmaster/pgstat.c b/src/backend/utils/activity/pgstat.c similarity index 99% rename from src/backend/postmaster/pgstat.c rename to src/backend/utils/activity/pgstat.c index d0315147bf1..dbd55a065d5 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -24,7 +24,7 @@ * Copyright (c) 2001-2022, PostgreSQL Global Development Group * * IDENTIFICATION - * src/backend/postmaster/pgstat.c + * src/backend/utils/activity/pgstat.c * ---------- */ #include "postgres.h" -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v68-0028-pgstat-update-docs.patch" From 0509ebb725efeff52ee0d89ada35e691c81c45a0 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sat, 2 Apr 2022 19:38:01 -0700 Subject: [PATCH v68 28/31] pgstat: update docs. ATODO: reflow comments intentionally not reflowed for easier review. --- doc/src/sgml/backup.sgml | 2 - doc/src/sgml/catalogs.sgml | 6 +- doc/src/sgml/config.sgml | 42 ++++++---- doc/src/sgml/func.sgml | 4 +- doc/src/sgml/glossary.sgml | 38 +++++---- doc/src/sgml/high-availability.sgml | 2 +- doc/src/sgml/maintenance.sgml | 2 +- doc/src/sgml/monitoring.sgml | 119 +++++++++++++++------------- doc/src/sgml/ref/pg_dump.sgml | 2 +- 9 files changed, 116 insertions(+), 101 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 0d69851bb1c..8cb7eb15ae7 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1185,8 +1185,6 @@ SELECT pg_stop_backup(); pg_snapshots/, pg_stat_tmp/, and pg_subtrans/ (but not the directories themselves) can be omitted from the backup as they will be initialized on postmaster startup. - If is set and is under the data - directory then the contents of that directory can also be omitted. diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 7f4f79d1b55..7a8905b849a 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -9530,9 +9530,9 @@ SCRAM-SHA-256$<iteration count>:&l lists the system views described here. More detailed documentation of each view follows below. - There are some additional views that provide access to the results of - the statistics collector; they are described in . + There are some additional views that provide access to accumulated + statistics; they are described in + . diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 43e4ade83e0..13ffd2e0e62 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -7795,12 +7795,12 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; Run-time Statistics - - Query and Index Statistics Collector + + Cumulative Query and Index Statistics - These parameters control server-wide statistics collection features. - When statistics collection is enabled, the data that is produced can be + These parameters control server-wide accumulated statistics system. + When enabled, the data that is collected can be accessed via the pg_stat and pg_statio family of system views. Refer to for more information. @@ -7935,22 +7935,36 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; - - stats_temp_directory (string) + + stats_fetch_consistency (enum) - stats_temp_directory configuration parameter + stats_fetch_consistency configuration parameter - Sets the directory to store temporary statistics data in. This can be - a path relative to the data directory or an absolute path. The default - is pg_stat_tmp. Pointing this at a RAM-based - file system will decrease physical I/O requirements and can lead to - improved performance. - This parameter can only be set in the postgresql.conf - file or on the server command line. + Determines the behaviour when cumulative statistics are accessed + multiple times within a transaction. When set to + none, each access re-fetches counters from shared + memory. When set to cache, the first access to + statistics for an object caches those statistics until the end of the + transaction / until pg_stat_clear_snapshot() is + called. When set to snapshot, the first statistics + access caches all statistics accessible in the current database, until + the end of the transaction / pg_stat_clear_snapshot() is + called. The default is cache. + + + none is most suitable for monitoring solutions. If + values are only accessed once, it is the most + efficient. cache ensures repeat accesses yield the + same values, which is important for queries involving + e.g. self-joins. snapshot can be useful when + interactively inspecting statistics, but has higher overhead, + particularly if many database objects exist. + + diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 4001cb2bda5..6fa9ac2ff88 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -25461,8 +25461,8 @@ SELECT collation for ('foo' COLLATE "de_DE"); Requests to log the memory contexts of the backend with the specified process ID. This function can send the request to - backends and auxiliary processes except logger and statistics - collector. These memory contexts will be logged at + backends and auxiliary processes except logger. These memory contexts + will be logged at LOG message level. They will appear in the server log based on the log configuration set (See for more information), diff --git a/doc/src/sgml/glossary.sgml b/doc/src/sgml/glossary.sgml index 1835d0e65a3..4c5e932acf9 100644 --- a/doc/src/sgml/glossary.sgml +++ b/doc/src/sgml/glossary.sgml @@ -136,9 +136,9 @@ The auxiliary processes consist of the autovacuum launcher (but not the autovacuum workers), @@ -146,7 +146,6 @@ the checkpointer, the logger, the startup process, - the statistics collector, the WAL archiver, the WAL receiver (but not the WAL senders), @@ -434,6 +433,21 @@ + + Cumulative Statistics System + + + A system which, if enabled, collects statistical information + about the instance's + activities. + + + For more information, see + . + + + + Data area @@ -1563,22 +1577,6 @@ - - Stats collector (process) - - - An auxiliary process - which, if enabled, receives statistical information - about the instance's - activities. - - - For more information, see - . - - - - System catalog diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml index 81fa26f9852..b3a307159ef 100644 --- a/doc/src/sgml/high-availability.sgml +++ b/doc/src/sgml/high-availability.sgml @@ -2227,7 +2227,7 @@ HINT: You can then restart the server after making the necessary configuration - The statistics collector is active during recovery. All scans, reads, blocks, + The cumulative statistics system is active during recovery. All scans, reads, blocks, index usage, etc., will be recorded normally on the standby. Replayed actions will not duplicate their effects on primary, so replaying an insert will not increment the Inserts column of pg_stat_user_tables. diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml index 2e09fee5aeb..a209a633043 100644 --- a/doc/src/sgml/maintenance.sgml +++ b/doc/src/sgml/maintenance.sgml @@ -839,7 +839,7 @@ vacuum insert threshold = vacuum base insert threshold + vacuum insert scale fac it may be beneficial to lower the table's as this may allow tuples to be frozen by earlier vacuums. The number of obsolete tuples and - the number of inserted tuples are obtained from the statistics collector; + the number of inserted tuples are obtained from the cumulative statistics system; it is a semi-accurate count updated by each UPDATE, DELETE and INSERT operation. (It is only semi-accurate because some information might be lost under heavy diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index 7c9469fb447..22d0a1e491f 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -22,7 +22,7 @@ Several tools are available for monitoring database activity and analyzing performance. Most of this chapter is devoted to describing - PostgreSQL's statistics collector, + PostgreSQL's cumulative statistics system, but one should not neglect regular Unix monitoring programs such as ps, top, iostat, and vmstat. Also, once one has identified a @@ -53,7 +53,6 @@ postgres 15554 0.0 0.0 57536 1184 ? Ss 18:02 0:00 postgres: back postgres 15555 0.0 0.0 57536 916 ? Ss 18:02 0:00 postgres: checkpointer postgres 15556 0.0 0.0 57536 916 ? Ss 18:02 0:00 postgres: walwriter postgres 15557 0.0 0.0 58504 2244 ? Ss 18:02 0:00 postgres: autovacuum launcher -postgres 15558 0.0 0.0 17512 1068 ? Ss 18:02 0:00 postgres: stats collector postgres 15582 0.0 0.0 58772 3080 ? Ss 18:04 0:00 postgres: joe runbug 127.0.0.1 idle postgres 15606 0.0 0.0 58772 3052 ? Ss 18:07 0:00 postgres: tgl regression [local] SELECT waiting postgres 15610 0.0 0.0 58772 3056 ? Ss 18:07 0:00 postgres: tgl regression [local] idle in transaction @@ -63,11 +62,10 @@ postgres 15610 0.0 0.0 58772 3056 ? Ss 18:07 0:00 postgres: tgl platforms, as do the details of what is shown. This example is from a recent Linux system.) The first process listed here is the primary server process. The command arguments - shown for it are the same ones used when it was launched. The next five + shown for it are the same ones used when it was launched. The next four processes are background worker processes automatically launched by the - primary process. (The stats collector process will not be present - if you have set the system not to start the statistics collector; likewise - the autovacuum launcher process can be disabled.) + primary process. (The autovacuum launcher process will not + be present if you have set the system not to run autovacuum.) Each of the remaining processes is a server process handling one client connection. Each such process sets its command line display in the form @@ -130,20 +128,20 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser - The Statistics Collector + The Cumulative Statistics System statistics - PostgreSQL's statistics collector - is a subsystem that supports collection and reporting of information about - server activity. Presently, the collector can count accesses to tables - and indexes in both disk-block and individual-row terms. It also tracks - the total number of rows in each table, and information about vacuum and - analyze actions for each table. It can also count calls to user-defined - functions and the total time spent in each one. + PostgreSQL's cumulative statistics + system supports collection and reporting of information about + server activity. Presently, accesses to tables and indexes in both + disk-block and individual-row terms are counted. The total number of rows + in each table, and information about vacuum and analyze actions for each + table are also counted. If enabled, calls to user-defined functions and + the total time spent in each one are counted as well. @@ -151,7 +149,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser information about exactly what is going on in the system right now, such as the exact command currently being executed by other server processes, and which other connections exist in the system. This facility is independent - of the collector process. + of the cumulative statistics system. @@ -172,7 +170,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser The parameter controls whether - statistics are collected about table and index accesses. + cumulative statistics are collected about table and index accesses. @@ -201,13 +199,9 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser - The statistics collector transmits the collected information to other - PostgreSQL processes through temporary files. - These files are stored in the directory named by the - parameter, - pg_stat_tmp by default. - For better performance, stats_temp_directory can be - pointed at a RAM-based file system, decreasing physical I/O requirements. + Cumulative statistics are collected in shared memory. Every + PostgreSQL process collects + statistics locally then updates the shared data at appropriate intervals. When the server shuts down cleanly, a permanent copy of the statistics data is stored in the pg_stat subdirectory, so that statistics can be retained across server restarts. When recovery is @@ -225,20 +219,20 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser linkend="monitoring-stats-dynamic-views-table"/>, are available to show the current state of the system. There are also several other views, listed in , available to show the results - of statistics collection. Alternatively, one can - build custom views using the underlying statistics functions, as discussed - in . + linkend="monitoring-stats-views-table"/>, available to show the accumulated + statistics. Alternatively, one can + build custom views using the underlying cumulative statistics functions, as + discussed in . - When using the statistics to monitor collected data, it is important + When using the accumulated statistics views and functions to monitor collected data, it is important to realize that the information does not update instantaneously. - Each individual server process transmits new statistical counts to - the collector just before going idle; so a query or transaction still in - progress does not affect the displayed totals. Also, the collector itself - emits a new report at most once per PGSTAT_STAT_INTERVAL - milliseconds (500 ms unless altered while building the server). So the + Each individual server process flushes out accumulated statistics to + shared memory just before going idle, but not more frequently than once + per PGSTAT_MIN_INTERVAL milliseconds (1 second unless + altered while building the server); so a query or transaction still in + progress does not affect the displayed totals and the displayed information lags behind actual activity. However, current-query information collected by track_activities is always up-to-date. @@ -246,27 +240,35 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser Another important point is that when a server process is asked to display - any of these statistics, it first fetches the most recent report emitted by - the collector process and then continues to use this snapshot for all - statistical views and functions until the end of its current transaction. - So the statistics will show static information as long as you continue the - current transaction. Similarly, information about the current queries of - all sessions is collected when any such information is first requested - within a transaction, and the same information will be displayed throughout - the transaction. - This is a feature, not a bug, because it allows you to perform several - queries on the statistics and correlate the results without worrying that - the numbers are changing underneath you. But if you want to see new - results with each query, be sure to do the queries outside any transaction - block. Alternatively, you can invoke - pg_stat_clear_snapshot(), which will discard the - current transaction's statistics snapshot (if any). The next use of - statistical information will cause a new snapshot to be fetched. + any of the accumulated statistics, acessed values are cached until the end + of its current transaction in the default configuration. So the statistics + will show static information as long as you continue the current + transaction. Similarly, information about the current queries of all + sessions is collected when any such information is first requested within a + transaction, and the same information will be displayed throughout the + transaction. This is a feature, not a bug, because it allows you to perform + several queries on the statistics and correlate the results without + worrying that the numbers are changing underneath you. + + When analyzing statistics interactively, or with expensive queries, the + time delta between accesses to indiviual statistics can lead to significant + skew in the cached statistics. To minimize skew, + stats_fetch_consistency can be set + snapshot, at the price of increased memory usage for + caching not-needed statistics data. Conversely, if known that statistics + are only accessed once, caching accessed statistics is unnecessary and can + be avoided by setting stats_fetch_consistency to + none. + + You can invoke pg_stat_clear_snapshot() to discard the + current transaction's statistics snapshot / cache (if any). The next use + of statistical information will cause a new snapshot to be built / accessed + statistics to be cached. - A transaction can also see its own statistics (as yet untransmitted to the - collector) in the views pg_stat_xact_all_tables, + A transaction can also see its own statistics (not yet flushed to the + shared statistics) in the views pg_stat_xact_all_tables, pg_stat_xact_sys_tables, pg_stat_xact_user_tables, and pg_stat_xact_user_functions. These numbers do not act as @@ -656,7 +658,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser kernel's I/O cache, and might therefore still be fetched without requiring a physical read. Users interested in obtaining more detailed information on PostgreSQL I/O behavior are - advised to use the PostgreSQL statistics collector + advised to use the PostgreSQL statistics views in combination with operating system utilities that allow insight into the kernel's handling of I/O. @@ -5094,7 +5096,7 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i - Additional functions related to statistics collection are listed in . @@ -5151,7 +5153,10 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i Returns the timestamp of the current statistics snapshot, or NULL if - no statistics snapshot has been taken. + no statistics snapshot has been taken. A snapshot is taken the first + time cumulative statistics are accessed in a transaction if + stats_fetch_consistency is set to + snapshot @@ -5164,7 +5169,7 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i void - Discards the current statistics snapshot. + Discards the current statistics snapshot or cached information. @@ -6325,8 +6330,8 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, VACUUM is performing final cleanup. During this phase, VACUUM will vacuum the free space map, update statistics - in pg_class, and report statistics to the statistics - collector. When this phase is completed, VACUUM will end. + in pg_class, and report statistics to the cumulative + statistics system. When this phase is completed, VACUUM will end. diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index 2f0042fd968..04839492777 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -1328,7 +1328,7 @@ PostgreSQL documentation The database activity of pg_dump is - normally collected by the statistics collector. If this is + normally collected by the cumulative statistics system. If this is undesirable, you can set parameter track_counts to false via PGOPTIONS or the ALTER USER command. -- 2.35.1.677.gabf474a5dd --c4n45bccvxafocvc--